Cited By
View all- Bang JKim CByun ESung HLee JEom H(2022)Accelerating I/O performance of ZFS-based Lustre file system in HPC environmentThe Journal of Supercomputing10.1007/s11227-022-04966-779:7(7665-7691)Online publication date: 5-Dec-2022
Provisioning ZFS Pools On Lustre
Authors: 
Rick Mohr, Adam P. HowardAuthors Info & Claims
Article No.: 52, Pages 1 - 6
Abstract
While Lustre's parallelism, performance, and scalability make it desirable as a storage solution for clusters, its limitations prevent it from being suitable as a general purpose storage for all of a cluster's needs. In particular, Lustre's relatively poor performance for small, random I/O as well as metadata-intensive workloads make it less suitable for use as storage for users' home areas or working directories for compilation. In this paper, an experiment of deploying a ZFS file system using files pre-allocated on a Lustre file system as the ZFS storage pools is presented. While this adds many management options like snapshots and more fine grained controls over users and quotas, we focus on examining how adding this layer can affect the performance of suboptimal Lustre workloads. Benchmarking of the Lustre file system with and without the ZFS file system is performed, and the results are presented and analyzed.
References
[1]
fio GitHub project. https://github.com/axboe/fio.
[2]
kcbench GitHub project. https://github.com/knurd/kcbench.
[3]
Linear Algebra PACKage. http://www.netlib.org/lapack.
[4]
Data on MDT Solution Architecture. http://wiki.lustre.org/Data_on_MDT_Solution_Architecture, May 2018.
[5]
Progressive File Layouts. http://wiki.lustre.org/Progressive_File_Layouts, June 2018.
[6]
OpenZFS. http://open-zfs.org/wiki/Main_Page, 2019.
Index Terms
- Provisioning ZFS Pools On Lustre
Recommendations
Accelerating I/O performance of ZFS-based Lustre file system in HPC environment
AbstractTo meet increasing data access performance demands of applications run on high-performance computing (HPC) systems, an efficient design of HPC storage file system is becoming more important. Nowadays, Lustre is the most commonly used distributed ...
NVMM-Oriented Hierarchical Persistent Client Caching for Lustre
Special Section on Usenix Fast 2020In high-performance computing (HPC), data and metadata are stored on special server nodes and client applications access the servers’ data and metadata through a network, which induces network latencies and resource contention. These server nodes are ...
Benchmarking SSD-Based Lustre File System Configurations
XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery EnvironmentDue to recent development efforts, ZFS on Linux is now a viable alternative to the traditional ldiskfs backend used for production Lustre file systems. Certain ZFS features, such as copy-on-write, make it even more appealing for systems utilizing SSD ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 July 2019
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
PEARC '19
PEARC '19: Practice and Experience in Advanced Research Computing
July 28 - August 1, 2019
IL, Chicago, USA
Acceptance Rates
Overall Acceptance Rate 133 of 202 submissions, 66%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 122Total Downloads
- Downloads (Last 12 months)15
- Downloads (Last 6 weeks)1
Reflects downloads up to 01 Nov 2024
Other Metrics
Citations
Cited By
View all- Bang JKim CByun ESung HLee JEom H(2022)Accelerating I/O performance of ZFS-based Lustre file system in HPC environmentThe Journal of Supercomputing10.1007/s11227-022-04966-779:7(7665-7691)Online publication date: 5-Dec-2022
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in