short-paper

Open access

HDF5 as a vehicle for in transit data movement

Authors:

Junmin Gu,

Burlen Loring,

Kesheng Wu,

E. Wes BethelAuthors Info & Claims

ISAV '19: Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization

Pages 39 - 43

https://doi.org/10.1145/3364228.3364237

Published: 18 November 2019 Publication History

PDF eReader

Abstract

For in transit processing, one of the fundamental challenges is the efficient movement of data from producers to consumers. Exploiting the flexibility offered by the SENSEI generic in situ framework, we have developed a number of different in transit data transport mechanisms. In this work, we focus on the transport mechanism that leverages the HDF5 parallel I/O library, and investigate the performance characteristics of this transport mechanism. For in transit use cases at scale on HPC platforms, one might expect that an in transit data transport mechanism that uses faster layers of the storage hierarchy, such as DRAM memory, would always outperform a transport that uses slower layers of the storage hierarchy, such as an NVRAM-based persistent storage presented as a distributed file system. However, our test results show that the performance of the transport using NVRAM is competitive with the transport that uses socket-based data movement across varying levels of producer and consumer concurrency.

References

[1]

U. Ayachit, A. Bauer, E. P. N. Duque, G. Eisenhauer, N. Ferrier, J. Gu, K. Jansen, B. Loring, Z. Lukić, S. Menon, D. Morozov, P. O'Leary, M. Rasquin, C. P. Stone, V. Vishwanath, G. H. Weber, B. Whitlock, M. Wolf, K. Wu, and E. W. Bethel. 2016. Performance Analysis, Design Considerations, and Applications of Extreme-scale In Situ Infrastructures. In ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC16). Salt Lake City, UT, USA. LBNL-1007264.

Crossref

Google Scholar

[2]

U. Ayachit, M. Whitlock, B. Wolf, B. Loring, B. Geveci, D. Lonie, and E. W. Bethel. 2016. The SENSEI Generic In Situ Interface. In Proceedings of In Situ Infrastructures for Enabling Extreme-scale Analysis and Visualization (ISAV 2016). Salt Lake City, UT, USA. LBNL-1007263.

Crossref

Google Scholar

[3]

W Bhimji, D Bard, M Romanus, D Paul, A Ovsyannikov, B Friesen, M Bryson, J Correa, G K Lockwood, V Tsulaia, S Byna, S Farrell, D Gursoy, C Daley, V Beckner, B Van Straalen, D Trebotich, C Tull, G Weber, N J Wright, K Antypas, and Prabhat. 2016. Accelerating Science with the NERSC Burst Buffer Early User Program. In CUG 2016. https://cug.org/proceedings/cug2016_proceedings/includes/files/pap162s2-file1.pdf

Google Scholar

[4]

J Dayal, D Bratcher, G Eisenhauer, K Schwan, M Wolf, X Zhang, H Abbasi, S Klasky, and N Podhorszki. 2014. Flexpath: Type-based publish/subscribe system for large-scale science analytics. In 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing. IEEE, 246--255.

Digital Library

Google Scholar

[5]

M Folk, G Heber, Q Koziol, E Pourmal, and D Robinson. 2011. An overview of the HDF5 technology suite and its applications. In Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases. ACM, 36--47. Software at http://www.hdfgroup.org/HDF5/.

Digital Library

Google Scholar

[6]

James A. Kohl, Torsten Wilde, and David E. Bernholdt. 2006. Cumulvs: Interacting with High-Performance Scientific Simulations, for Visualization, Steering and Fault Tolerance. The International Journal of High Performance Computing Applications 20, 2 (2006), 255--285.

Digital Library

Google Scholar

[7]

J. Kress, M. Larsen, J. Choi, M. Kim, M. Wolf, N. Podhorszki, S. Klasky, H. Childs, and D. Pugmire. 2019. Comparing the Efficiency of In Situ Visualization Paradigms at Scale. In High Performance Computing, Michèle Weiland, Guido Juckeland, Carsten Trinitis, and Ponnuswamy Sadayappan (Eds.). Springer International Publishing, Cham, 99--117.

Google Scholar

[8]

Q. Liu, J. Logan, Y. Tian, H. Abbasi, N. Podhorszki, Jong Y. Choi, S. Klasky, R. Tchoua, J. Lofstead, R. Oldfield, M. Parashar, N. Samatova, K. Schwan, A. Shoshani, M. Wolf, K. Wu, and W. Yu. 2014. Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. Concurrency and Computation: Practice and Experience 26, 7 (2014), 1453--1473.

Digital Library

Google Scholar

[9]

W Usher, S Rizzi, I Wald, J Amstutz, Jh Insley, V Vishwanath, N Ferrier, M E Papka, and V Pascucci. 2018. libIS: A Lightweight Library for Flexible in Transit Visualization. In Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV '18). ACM, New York, NY, USA, 33--38.

Digital Library

Google Scholar

Cited By

View all

Loring BBethel EWeber GMahoney M(2023)Extensions to the SENSEI In situ Framework for Heterogeneous ArchitecturesProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624161(868-874)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624161
Bethel ELoring BAyachit UDuque EFerrier NInsley JGu JKress JO’Leary PPugmire DRizzi SThompson DUsher WWeber GWhitlock BWolf MWu K(2022)Proximity Portability and in Transit, M-to-N Data Partitioning and Movement in SENSEIIn Situ Visualization for Computational Science10.1007/978-3-030-81627-8_20(439-460)Online publication date: 5-May-2022
https://doi.org/10.1007/978-3-030-81627-8_20
Walldén MOkita MIno FDrikakis DKokkinakis I(2021)Accelerating In-Transit Co-Processing for Scientific Simulations Using Region-Based Data-Driven AnalysisAlgorithms10.3390/a1405015414:5(154)Online publication date: 12-May-2021
https://doi.org/10.3390/a14050154

Index Terms

HDF5 as a vehicle for in transit data movement

Recommendations

Evaluating rail transit timetable using big passengers' data

The unpredicted and unbalanced nature of passenger flow over temporal and spatial complicates the design and optimization of scheduled timetables. This study presents a model based on time-driven microscopic simulation to evaluate train timetable from ...
libIS: a lightweight library for flexible in transit visualization
ISAV '18: Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization

As simulations grow in scale, the need for in situ analysis methods to handle the large data produced grows correspondingly. One desirable approach to in situ visualization is in transit visualization. By decoupling the simulation and visualization code,...
Collinear Competition between Rail Transit and Bus Transit Based on Population Competition Model
ICIE '10: Proceedings of the 2010 WASE International Conference on Information Engineering - Volume 03

The paper discusses the collinear competition problem between urban rail transit and conventional bus transit. It compares ecological population competition with collinear transit competition, and utilizes ecological theories and means to solve the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ISAV '19: Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization

November 2019

56 pages

ISBN:9781450377232

DOI:10.1145/3364228

General Chairs:
Ken Moreland
Sandia National Laboratory
,
Christoph Garth
Technische Universität Kaiserslautern, Germany
,
Program Chairs:
E. Wes Bethel
Lawrence Berkeley National Laboratory
,
Bruno Raffin
INRIA, France

© 2019 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the United States Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 November 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

US Department of Energy

Conference

ISAV'19

ISAV'19: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization

November 18, 2019

Colorado, Denver, USA

Acceptance Rates

Overall Acceptance Rate 23 of 63 submissions, 37%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
306
Total Downloads

Downloads (Last 12 months)99
Downloads (Last 6 weeks)16

Reflects downloads up to 09 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Loring BBethel EWeber GMahoney M(2023)Extensions to the SENSEI In situ Framework for Heterogeneous ArchitecturesProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624161(868-874)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624161
Bethel ELoring BAyachit UDuque EFerrier NInsley JGu JKress JO’Leary PPugmire DRizzi SThompson DUsher WWeber GWhitlock BWolf MWu K(2022)Proximity Portability and in Transit, M-to-N Data Partitioning and Movement in SENSEIIn Situ Visualization for Computational Science10.1007/978-3-030-81627-8_20(439-460)Online publication date: 5-May-2022
https://doi.org/10.1007/978-3-030-81627-8_20
Walldén MOkita MIno FDrikakis DKokkinakis I(2021)Accelerating In-Transit Co-Processing for Scientific Simulations Using Region-Based Data-Driven AnalysisAlgorithms10.3390/a1405015414:5(154)Online publication date: 12-May-2021
https://doi.org/10.3390/a14050154

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations

Evaluating rail transit timetable using big passengers' data

libIS: a lightweight library for flexible in transit visualization

Collinear Competition between Rail Transit and Bus Transit Based on Population Competition Model

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations