research-article

Open access

Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context

Authors:

Ranjan Sarpangala Venkatesh,

Greg Eisenhauer,

Scott Klasky,

Ada GavrilovskaAuthors Info & Claims

SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

Pages 1223 - 1228

https://doi.org/10.1145/3624062.3624193

Published: 12 November 2023 Publication History

All formats PDF

Abstract

In HPC, data movement between applications is typically facilitated by I/O middleware, such as the Adaptable I/O System (ADIOS). This middleware leverages the capabilities of the underlying storage services, to facilitate data movement and distribution. A recent storage system, Intel DAOS, promises to deliver new capabilities for achieving performance and scalability on emerging memory/storage systems. DAOS has already deployed in University of Cambridge, TACC, and the upcoming Aurora supercomputer. This paper investigates the performance tradeoffs associated with mapping ADIOS over one of the many different DAOS interfaces and data models, and makes recommendations for their efficient use.

Supplemental Material

MP4 File

Recording of "Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context" presentation at PDSW 2023.

Download
117.10 MB

References

[1]

Sadaf R Alam, Hussein N El-Harake, Kristopher Howard, Neil Stringfellow, and Fabio Verzelloni. 2011. Parallel I/O and the metadata wall. In Proceedings of the sixth workshop on Parallel Data Storage. 13–18.

Digital Library

Google Scholar

[2]

Michael Hennecke. 2020. Daos: A scale-out high performance storage stack for storage class memory. Supercomputing frontiers (2020), 40.

Google Scholar

[3]

Jialin Liu, Quincey Koziol, Gregory F. Butler, Neil Fortner, Mohamad Chaarawi, Houjun Tang, Suren Byna, Glenn K. Lockwood, Ravi Cheema, Kristy A. Kallback-Rose, Damian Hazen, and Mr Prabhat. 2018. Evaluation of HPC Application I/O on Object Storage Systems. In 2018 IEEE/ACM 3rd International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems (PDSW-DISCS). 24–34. https://doi.org/10.1109/PDSW-DISCS.2018.00005

Crossref

Google Scholar

[4]

Qing Liu, Jeremy Logan, Yuan Tian, Hasan Abbasi, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, Roselyne Tchoua, Jay Lofstead, Ron Oldfield, 2014. Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. Concurrency and Computation: Practice and Experience 26, 7 (2014), 1453–1473.

Digital Library

Google Scholar

[5]

Ricardo Macedo, Mariana Miranda, Yusuke Tanimura, Jason Haga, Amit Ruhela, Stephen Lien Harrell, Richard Todd Evans, José Pereira, and Joao Paulo. 2023. Taming Metadata-intensive HPC Jobs Through Dynamic, Application-agnostic QoS Control. In 23nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid). IEEE.

Google Scholar

[6]

Jerome Soumagne, Jordan Henderson, Mohamad Chaarawi, Neil Fortner, Scot Breitenfeld, Songyu Lu, Dana Robinson, Elena Pourmal, and Johann Lombardi. 2021. Accelerating hdf5 i/o for exascale using daos. IEEE Transactions on Parallel and Distributed Systems 33, 4 (2021), 903–914.

Crossref

Google Scholar

[7]

Ranjan Sarpangala Venkatesh, Tony Mason, Pradeep Fernando, Greg Eisenhauer, and Ada Gavrilovska. 2021. Scheduling HPC Workflows with Intel Optane Persistent Memory. In 2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 56–65.

Google Scholar

Cited By

View all

Manubens NSmart SDanovaro EQuintino TJackson AHuebl ASilvano CRobinson TWoodtli M(2024)Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOSProceedings of the Platform for Advanced Scientific Computing Conference10.1145/3659914.3659926(1-12)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3659914.3659926

Index Terms

Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context
1. Information systems
  1. Information storage systems
    1. Information storage technologies
      1. Storage class memory
        Flash memory
        Phase change memory
    2. Storage architectures
      1. Distributed storage

Recommendations

Data management services and transfer scheme in ChinaGrid

ChinaGrid is a nationwide grid project supported by the Chinese government that aims at integrating massive heterogeneous resources distributed on the China Education and Research Network (CERNET). The ChinaGrid Supporting Platform (CGSP) is a grid ...
Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks

Applications running on leadership platforms are more and more bottlenecked by storage input/output I/O. In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System ADIOS in 2005. ...
UNICORE in XSEDE: Through Development, Integration, Deployment and beyond
XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment

In this paper we discuss the implementation of UNICORE in XSEDE. UNICORE is a Grid middleware tool that was identified by XSEDE to further the areas of remote job submission, campus bridging and workflows. We talk about the overall architecture of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

November 2023

2180 pages

ISBN:9798400707858

DOI:10.1145/3624062

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SC-W 2023

SC-W 2023: Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

November 12 - 17, 2023

CO, Denver, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
216
Total Downloads

Downloads (Last 12 months)216
Downloads (Last 6 weeks)41

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Manubens NSmart SDanovaro EQuintino TJackson AHuebl ASilvano CRobinson TWoodtli M(2024)Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOSProceedings of the Platform for Advanced Scientific Computing Conference10.1145/3659914.3659926(1-12)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3659914.3659926

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

Supplemental Material

References

Cited By

Index Terms

Recommendations

Data management services and transfer scheme in ChinaGrid

Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks

UNICORE in XSEDE: Through Development, Integration, Deployment and beyond

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

HTML Format

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations