Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3579370.3594769acmconferencesArticle/Chapter ViewAbstractPublication PagessystorConference Proceedingsconference-collections
short-paper
Open access

DPFS: DPU-Powered File System Virtualization

Published: 22 June 2023 Publication History

Abstract

As we move towards hyper-converged cloud solutions, the efficiency and overheads of distributed file systems at the cloud tenant side (i.e., client) become of paramount importance. Often, the clientside driver of a cloud file system is complex and CPU intensive, deeply coupled with the backend implementation, and requires optimizing multiple intrusive knobs. In this work, we propose to decouple the file system client from its backend implementation by virtualizing it with an off-the-shelf DPU using the Linux virtio-fs software stack. The decoupling allows us to offload the file system client execution to a DPU, which is managed and optimized by the cloud provider, while freeing the host CPU cycles. DPFS, our proposed framework, is 4.4× more host CPU efficient per I/O, delivers comparable performance to a tenant with zero-configuration and without modification of their host software stack, while allowing workload and hardware specific backend optimizations. The DPFS framework and its artifacts are publically available at https://github.com/IBM/DPFS.

References

[1]
2023. Libfuse: The Reference Implementation of the Linux FUSE (Filesystem in Userspace) Interface. https://github.com/libfuse/libfuse
[2]
Alexandru Agache, Marc Brooker, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and Diana-Maria Popa. 2020. Firecracker: Lightweight Virtualization for Serverless Applications. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). USENIX Association, Santa Clara, CA, 419--434. https://www.usenix.org/conference/nsdi20/presentation/agache
[3]
Amazon AWS. 2023. What is Amazon Elastic File System? - Amazon Elastic File System. https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html
[4]
Brad Burres, Dan Daly, Mark Debbage, Eliel Louzoun, Christine Severns-Williams, Naru Sundar, Nadav Turbovich, Barry Wolford, and Yadong Li. 2021. Intel's Hyperscale-Ready Infrastructure Processing Unit (IPU). In 2021 IEEE Hot Chips 33 Symposium (HCS). 1--16.
[5]
Idan Burstein. 2021. Nvidia Data Center Processing Unit (DPU) Architecture. In 2021 IEEE Hot Chips 33 Symposium (HCS). 1--20.
[6]
Christina Delimitrou and Christos Kozyrakis. 2017. Bolt: I Know What You Did Last Summer... In The Cloud. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems (Xi'an, China) (ASPLOS '17). Association for Computing Machinery, New York, NY, USA, 599--613.
[7]
Diego Didona, Jonas Pfefferle, Nikolas Ioannou, Bernard Metzler, and Animesh Trivedi. 2022. Understanding Modern Storage APIs: A Systematic Study of Libaio, SPDK, and Io_uring. In Proceedings of the 15th ACM International Conference on Systems and Storage (Haifa, Israel) (SYSTOR '22). Association for Computing Machinery, New York, NY, USA, 120--127.
[8]
Yaozu Dong, Xiaowei Yang, Xiaoyong Li, Jianhui Li, Kun Tian, and Haibing Guan. 2010. High performance network virtualization with SR-IOV. In HPCA - 16 2010 The Sixteenth International Symposium on High-Performance Computer Architecture. 1--10.
[9]
Daniel Firestone, Andrew Putnam, Sambhrama Mundkur, Derek Chiou, Alireza Dabagh, Mike Andrewartha, Hari Angepat, Vivek Bhanu, Adrian Caulfield, Eric Chung, Harish Kumar Chandrappa, Somesh Chaturmohta, Matt Humphrey, Jack Lavier, Norman Lam, Fengfen Liu, Kalin Ovtcharov, Jitu Padhye, Gautham Popuri, Shachar Raindel, Tejas Sapre, Mark Shaw, Gabriel Silva, Madhan Sivakumar, Nisheeth Srivastava, Anshuman Verma, Qasim Zuhair, Deepak Bansal, Doug Burger, Kushagra Vaid, David A. Maltz, and Albert Greenberg. 2018. Azure Accelerated Networking: SmartNICs in the Public Cloud. In Proceedings of the 15th USENIX Conference on Networked Systems Design and Implementation (Renton, WA, USA) (NSDI'18). USENIX Association, USA, 51--64.
[10]
Yixiao Gao, Qiang Li, Lingbo Tang, Yongqing Xi, Pengcheng Zhang, Wenwen Peng, Bo Li, Yaohui Wu, Shaozong Liu, Lei Yan, Fei Feng, Yan Zhuang, Fan Liu, Pan Liu, Xingkui Liu, Zhongjie Wu, Junping Wu, Zheng Cao, Chen Tian, Jinbo Wu, Jiaji Zhu, Haiyong Wang, Dennis Cai, and Jiesheng Wu. 2021. When Cloud Storage Meets RDMA. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). USENIX Association, 519--533. https://www.usenix.org/conference/nsdi21/presentation/gao
[11]
Google. 2021. Colossus under the hood: a peek into Google's scalable storage system. https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system
[12]
Google Cloud. 2023. Technical overview - Google Cloud Filestore. https://cloud.google.com/filestore/docs/overview
[13]
Karan Gupta. 2020. From Hyper Converged Infrastructure to Hybrid Cloud Infrastructure. USENIX Association.
[14]
Jaehyun Hwang, Qizhe Cai, Ao Tang, and Rachit Agarwal. 2020. TCP ≈ RDMA: CPU-efficient Remote Storage Access with i10. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). USENIX Association, Santa Clara, CA, 127--140. https://www.usenix.org/conference/nsdi20/presentation/hwang
[15]
Junbin Kang, Chunming Hu, Tianyu Wo, Ye Zhai, Benlong Zhang, and Jinpeng Huai. 2016. MultiLanes: Providing Virtualized Storage for OS-Level Virtualization on Manycores. ACM Trans. Storage 12, 3, Article 12 (jun 2016), 31 pages.
[16]
Jongyul Kim, Insu Jang, Waleed Reda, Jaeseong Im, Marco Canini, Dejan Kostić, Youngjin Kwon, Simon Peter, and Emmett Witchel. 2021. LineFS: Efficient Smart-NIC Offload of a Distributed File System with Pipeline Parallelism. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles (Virtual Event, Germany) (SOSP '21). Association for Computing Machinery, New York, NY, USA, 756--771.
[17]
Ana Klimovic, Heiner Litz, and Christos Kozyrakis. 2017. ReFlex: Remote Flash ≈ Local Flash. SIGARCH Comput. Archit. News 45, 1 (apr 2017), 345--359.
[18]
Jinhyung Koo, Junsu Im, Jooyoung Song, Juhyung Park, Eunji Lee, Bryan S. Kim, and Sungjin Lee. 2021. Modernizing File System through In-Storage Indexing. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21). USENIX Association, 75--92. https://www.usenix.org/conference/osdi21/presentation/koo
[19]
Chunbo Lai, Song Jiang, Liqiong Yang, Shiding Lin, Guangyu Sun, Zhenyu Hou, Can Cui, and Jason Cong. 2015. Atlas: Baidu's key-value storage system for cloud data. In 2015 31st Symposium on Mass Storage Systems and Technologies (MSST). 1--14.
[20]
Huaicheng Li, Mingzhe Hao, Stanko Novakovic, Vaibhav Gogte, Sriram Govindan, Dan R. K. Ports, Irene Zhang, Ricardo Bianchini, Haryadi S. Gunawi, and Anirudh Badam. 2020. LeapIO: Efficient and Portable Virtual NVMe Storage on ARM SoCs. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (Lausanne, Switzerland) (ASPLOS '20). Association for Computing Machinery, New York, NY, USA, 591--605.
[21]
Qiang Li, Lulu Chen, Xiaoliang Wang, Shuo Huang, Qiao Xiang, Yuanyuan Dong, Wenhui Yao, Minfei Huang, Puyuan Yang, Shanyang Liu, Zhaosheng Zhu, Huayong Wang, Haonan Qiu, Derui Liu, Shaozong Liu, Yujie Zhou, Yaohui Wu, Zhiwu Wu, Shang Gao, Chao Han, Zicheng Luo, Yuchao Shao, Gexiao Tian, Zhongjie Wu, Zheng Cao, Jinbo Wu, Jiwu Shu, Jie Wu, and Jiesheng Wu. 2023. Fisc: A Large-Scale Cloud-Native-Oriented File System. In Proceedings of the 21st USENIX Conference on File and Storage Technologies (Santa Clara, CA, USA) (FAST'23). USENIX Association, USA, Article 15, 15 pages.
[22]
Kunal Lillaney, Vasily Tarasov, David Pease, and Randal Burns. 2019. Agni: An Efficient Dual-Access File System over Object Storage. In Proceedings of the ACM Symposium on Cloud Computing (Santa Cruz, CA, USA) (SoCC '19). Association for Computing Machinery, New York, NY, USA, 390--402.
[23]
Ming Liu, Simon Peter, Arvind Krishnamurthy, and Phitchaya Mangpo Phothilimthana. 2019. E3: Energy-Efficient Microservices on SmartNIC-Accelerated Servers. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). USENIX Association, Renton, WA, 363--378. https://www.usenix.org/conference/atc19/presentation/liu-ming
[24]
Youyou Lu, Jiwu Shu, and Weimin Zheng. 2013. Extending the Lifetime of Flash-Based Storage through Reducing Write Amplification from File Systems. In Proceedings of the 11th USENIX Conference on File and Storage Technologies (San Jose, CA) (FAST'13). USENIX Association, USA, 257--270.
[25]
Wenhao Lv, Youyou Lu, Yiming Zhang, Peile Duan, and Jiwu Shu. 2022. InfiniFS: An Efficient Metadata Service for Large-Scale Distributed Filesystems. In 20th USENIX Conference on File and Storage Technologies (FAST 22). USENIX Association, Santa Clara, CA, 313--328. https://www.usenix.org/conference/fast22/presentation/lv
[26]
Rui Miao, Lingjun Zhu, Shu Ma, Kun Qian, Shujun Zhuang, Bo Li, Shuguang Cheng, Jiaqi Gao, Yan Zhuang, Pengcheng Zhang, Rong Liu, Chao Shi, Binzhang Fu, Jiaji Zhu, Jiesheng Wu, Dennis Cai, and Hongqiang Harry Liu. 2022. From Luna to Solar: The Evolutions of the Compute-to-Storage Networks in Alibaba Cloud. In Proceedings of the ACM SIGCOMM 2022 Conference (Amsterdam, Netherlands) (SIGCOMM '22). Association for Computing Machinery, New York, NY, USA, 753--766.
[27]
Microsoft Azure. 2023. Azure Files - Managed File Shares and Storage. https://azure.microsoft.com/en-us/products/storage/files
[28]
Jaehong Min, Ming Liu, Tapan Chugh, Chenxingyu Zhao, Andrew Wei, In Hwan Doh, and Arvind Krishnamurthy. 2021. Gimbal: Enabling Multi-Tenant Storage Disaggregation on SmartNIC JBOFs. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference (Virtual Event, USA) (SIGCOMM '21). Association for Computing Machinery, New York, NY, USA, 106--122.
[29]
Mihir Nanavati, Malte Schwarzkopf, Jake Wires, and Andrew Warfield. 2015. Non-Volatile Storage: Implications of the Datacenter's Shifting Center. Queue 13, 9 (nov 2015), 33--56.
[30]
Salman Niazi, Mahmoud Ismail, Seif Haridi, Jim Dowling, Steffen Grohsschmiedt, and Mikael Ronström. 2017. HopsFS: Scaling Hierarchical File System Metadata Using NewSQL Databases. In 15th USENIX Conference on File and Storage Technologies (FAST 17). USENIX Association, Santa Clara, CA, 89--104. https://www.usenix.org/conference/fast17/technical-sessions/presentation/niazi
[31]
David Noveck and Chuck Lever. 2020. Network File System (NFS) Version 4 Minor Version 1 Protocol. RFC 8881.
[32]
NVIDIA. 2021. NVIDIA BlueField-2 Datasheet. https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/documents/datasheet-nvidia-bluefield-2-dpu.pdf
[33]
NVIDIA. 2023. NVIDIA Accelerated IO (XLIO) Documentation. https://docs.nvidia.com/networking/display/XLIOv214
[34]
Yushi Omote, Takahiro Shinagawa, and Kazuhiko Kato. 2015. Improving Agility and Elasticity in Bare-Metal Clouds. SIGARCH Comput. Archit. News 43, 1 (mar 2015), 145--159.
[35]
John Ousterhout, Arjun Gopalan, Ashish Gupta, Ankita Kejriwal, Collin Lee, Behnam Montazeri, Diego Ongaro, Seo Jin Park, Henry Qin, Mendel Rosenblum, Stephen Rumble, Ryan Stutsman, and Stephen Yang. 2015. The RAMCloud Storage System. ACM Trans. Comput. Syst. 33, 3, Article 7 (aug 2015), 55 pages.
[36]
Satadru Pan, Theano Stavrinos, Yunqiao Zhang, Atul Sikaria, Pavel Zakharov, Abhinav Sharma, Shiva Shankar P, Mike Shuey, Richard Wareing, Monika Gangapuram, Guanglei Cao, Christian Preseau, Pratap Singh, Kestutis Patiejunas, JR Tipton, Ethan Katz-Bassett, and Wyatt Lloyd. 2021. Facebook's Tectonic Filesystem: Efficiency from Exascale. In 19th USENIX Conference on File and Storage Technologies (FAST 21). USENIX Association, 217--231. https://www.usenix.org/conference/fast21/presentation/pan
[37]
Red Hat, Inc. 2022. virtio-fs Linux Kernel implementation - Linux kernel source tree v6.2. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/fuse/virtio_fs.c?h=v6.2
[38]
Rusty Russell. 2008. Virtio: Towards a de-Facto Standard for Virtual I/O Devices. SIGOPS Oper. Syst. Rev. 42, 5 (jul 2008), 95--103.
[39]
Ronnie Sahlberg. 2023. Libnfs: NFS client library. https://github.com/sahlberg/libnfs
[40]
Giuseppe Siracusano and Roberto Bifulco. 2017. Is It a SmartNIC or a Key-Value Store? Both!. In Proceedings of the SIGCOMM Posters and Demos (Los Angeles, CA, USA) (SIGCOMM Posters and Demos '17). Association for Computing Machinery, New York, NY, USA, 138--140.
[41]
Brent Stephens, Aditya Akella, and Michael M. Swift. 2018. Your Programmable NIC Should Be a Programmable Switch. In Proceedings of the 17th ACM Workshop on Hot Topics in Networks (Redmond, WA, USA) (HotNets '18). Association for Computing Machinery, New York, NY, USA, 36--42.
[42]
Maroun Tork, Lina Maudlej, and Mark Silberstein. 2020. Lynx: A SmartNIC-Driven Accelerator-Centric Architecture for Network Servers. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (Lausanne, Switzerland) (ASPLOS '20). Association for Computing Machinery, New York, NY, USA, 117--131.
[43]
Animesh Trivedi, Nikolas Ioannou, Bernard Metzler, Patrick Stuedi, Jonas Pfefferle, Kornilios Kourtis, Ioannis Koltsidas, and Thomas R. Gross. 2018. FlashNet: Flash/Network Stack Co-Design. ACM Trans. Storage 14, 4, Article 30 (dec 2018), 29 pages.
[44]
Animesh Trivedi, Patrick Stuedi, Jonas Pfefferle, Adrian Schuepbach, and Bernard Metzler. 2018. Albis: High-Performance File Format for Big Data Systems. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). USENIX Association, Boston, MA, 615--630. https://www.usenix.org/conference/atc18/presentation/trivedi
[45]
Shin-Yeh Tsai, Yizhou Shan, and Yiying Zhang. 2020. Disaggregating Persistent Memory and Controlling Them Remotely: An Exploration of Passive Disaggregated Key-Value Stores. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). USENIX Association, 33--48. https://www.usenix.org/conference/atc20/presentation/tsai
[46]
Michael S. Tsirkin and Cornelia Huck. 2022. Virtual I/O Device (VIRTIO) Version 1.2. https://docs.oasis-open.org/virtio/virtio/v1.2/virtio-v1.2.html
[47]
Bharath Kumar Reddy Vangoor, Vasily Tarasov, and Erez Zadok. 2017. To FUSE or Not to FUSE: Performance of User-Space File Systems. In 15th USENIX Conference on File and Storage Technologies (FAST 17). USENIX Association, Santa Clara, CA, 59--72. https://www.usenix.org/conference/fast17/technical-sessions/presentation/vangoor
[48]
Xiantao Zhang, Xiao Zheng, Zhi Wang, Hang Yang, Yibin Shen, and Xin Long. 2020. High-Density Multi-Tenant Bare-Metal Cloud. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (Lausanne, Switzerland) (ASPLOS '20). Association for Computing Machinery, New York, NY, USA, 483--495.
[49]
Zhipeng Zhao, Hugo Sadok, Nirav Atre, James C. Hoe, Vyas Sekar, and Justine Sherry. 2020. Achieving 100Gbps Intrusion Prevention on a Single Server. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI'20). USENIX Association, USA, Article 61, 18 pages.

Cited By

View all
  • (2024)Cheap & Fast File-aaS for AI by Combining Scale-out Virtiofs, Block layouts and DelegationsProceedings of the 17th ACM International Systems and Storage Conference10.1145/3688351.3689167(194-195)Online publication date: 16-Sep-2024
  • (2024)DPC: DPU-accelerated High-Performance File System ClientProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673123(63-72)Online publication date: 12-Aug-2024
  • (2024)D2Comp: Efficient Offload of LSM-tree Compaction with Data Processing Units on Disaggregated StorageACM Transactions on Architecture and Code Optimization10.1145/365658421:3(1-22)Online publication date: 9-Apr-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
SYSTOR '23: Proceedings of the 16th ACM International Conference on Systems and Storage
June 2023
168 pages
ISBN:9781450399623
DOI:10.1145/3579370
This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 June 2023

Check for updates

Author Tags

  1. DPU
  2. SmartNIC
  3. offloading
  4. file system
  5. virtualization
  6. cloud
  7. storage
  8. framework
  9. datacenter
  10. RDMA
  11. NFS
  12. virtio-fs
  13. FUSE

Qualifiers

  • Short-paper

Funding Sources

Conference

SYSTOR '23
Sponsor:

Acceptance Rates

SYSTOR '23 Paper Acceptance Rate 12 of 30 submissions, 40%;
Overall Acceptance Rate 108 of 323 submissions, 33%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1,186
  • Downloads (Last 6 weeks)127
Reflects downloads up to 10 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Cheap & Fast File-aaS for AI by Combining Scale-out Virtiofs, Block layouts and DelegationsProceedings of the 17th ACM International Systems and Storage Conference10.1145/3688351.3689167(194-195)Online publication date: 16-Sep-2024
  • (2024)DPC: DPU-accelerated High-Performance File System ClientProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673123(63-72)Online publication date: 12-Aug-2024
  • (2024)D2Comp: Efficient Offload of LSM-tree Compaction with Data Processing Units on Disaggregated StorageACM Transactions on Architecture and Code Optimization10.1145/365658421:3(1-22)Online publication date: 9-Apr-2024
  • (2024)Efficient Deployment of Partial Parallelized Service Function Chains in CPU+DPU-Based Heterogeneous NFV PlatformsIEEE Transactions on Mobile Computing10.1109/TMC.2024.335779623:10(9090-9107)Online publication date: Oct-2024
  • (2024) Reviving Storage Systems Education in the 21 st Century — An experience report 2024 IEEE 24th International Symposium on Cluster, Cloud and Internet Computing (CCGrid)10.1109/CCGrid59990.2024.00074(616-625)Online publication date: 6-May-2024
  • (2023)DComp: Efficient Offload of LSM-tree Compaction with Data Processing UnitsProceedings of the 52nd International Conference on Parallel Processing10.1145/3605573.3605633(233-243)Online publication date: 7-Aug-2023
  • (2023)CPU-free Computing: A Vision with a BlueprintProceedings of the 19th Workshop on Hot Topics in Operating Systems10.1145/3593856.3595906(1-14)Online publication date: 22-Jun-2023

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media