MultIO: A Framework for Message-Driven Data Routing For Weather and Climate Simulations
Authors: 
Domokos Sarmany, Mirco Valentini, Pedro Maciel, Philipp Geier, Simon Smart, Razvan Aguridan, James Hawkes, Tiago QuintinoAuthors Info & Claims
Article No.: 24, Pages 1 - 12
Abstract
In numerical weather prediction and high-performance computing, the primary computational bottleneck has gradually evolved from floating-point arithmetic to the throughput of data to and from the storage. This phenomenon is commonly referred to as the I/O performance gap. We present MultIO, a set of software libraries that provide two mechanisms to mitigate this effect: an asynchronous I/O-server to decouple data output from model computations, and user-programmable processing pipelines that operate on model output directly.
MultIO is a metadata-driven, message-based system. This means that the I/O-server and processing pipelines fundamentally handle and operate on discrete self-describing messages. The behaviour of the I/O-server, data routing decisions and selection of actions undertaken are driven by the metadata attached to each message. The user may control the type and amount of post-processing by setting the message metadata via the Fortran/C/Python APIs, and by configuring a processing pipeline of actions. Users are also able to implement custom actions to be incorporated into the pipelines.
The MultIO system has been used with the NEMOv4 ocean model to implement the upcoming ocean re-analysis dataset, which will feed into the production runs of the next generation of global re-analysis dataset, ERA6. It has also been used to move computation closer to the model for climate runs at scale in two projects funded by the European Union: nextGEMS and Destination Earth.
References
[1]
[SW Rel.], version 2.0.2, 2023. url: https://github.com/ecmwf/multio/.
[2]
S. Danilov, D. Sidorenko, Q. Wang, and T. Jung. 2017. The finite-volume sea ice-ocean model (fesom2). Geoscientific Model Development, 10, 2, 765--789.
[3]
European Centre for Medium-Range Weather Forecasts. [n. d.] Integrated forecasting system. https://www.ecmwf.int/en/publications/ifs-documentation.
[4]
William F. Godoy et al. 2020. Adios 2: the adaptable input output system. a framework for high-performance data management. SoftwareX, 12, 100561.
[5]
K. M. Górski, E. Hivon, A. J. Banday, B. D. Wandelt, F. K. Hansen, M. Reinecke, and M. Bartelmann. 2005. Healpix: a framework for high-resolution discretization and fast analysis of data distributed on the sphere. The Astrophysical Journal, 622, 2, (Apr. 2005), 759.
[6]
Madec Gurvan et al. 2020. Nemo ocean engine. Version v4.0. (June 2020).
[7]
Christopher Haine, Utz-Uwe Haus, Maxime Martinasso, Dirk Pleiter, François Tessier, Domokos Sármány, Simon Smart, Tiago Quintino, and Adrian Tate. 2021. A middleware supporting data movement in complex and software-defined storage and memory architectures. In High Performance Computing. Heike Jagode, Hartwig Anzt, Hatem Ltaief, and Piotr Luszczek, (Eds.) Springer International Publishing, Cham, 346--357. isbn: 978-3-030-90539-2.
[8]
Hans Hersbach et al. 2020. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146, 730, 1999--2049.
[9]
Message Passing Interface Forum. 2023. MPI: A Message-Passing Interface Standard Version 4.1. (Nov. 2023). https://www.mpi-forum.org/docs/mpi-4.1/mpi41-report.pdf.
[10]
Y. Meurdesoif, A. Caubel, R. Lacroix, J. Dérouillat, and M.H. Nguye. 2016. XIOS Tutorial. (Jan. 2016). Retrieved Jan 14, 2016 from https://data-croco.ifremer.fr/DOC/PRES_TRAINING_CROCO_2016/10_12_XIOS-tutorial.pdf.
[11]
Y. Meurdesoif, J. Dérouillat, and A. Caubel. 2023. XIOS-3 Toward a new infrastructure of HPC services and model coupling. (Jan. 2023). Retrieved Jan 20, 2023 from https://forge.ipsl.jussieu.fr/ioserver/raw-attachment/wiki/WikiStart/Coupling%20workshop-CW2023.pdf.
[12]
World Meteorological Organization. 2019. Manual on Codes. World Meteorological Organization. https://library.wmo.int/idurl/4/35713.
[13]
Jaideep Pathak et al. 2022. Fourcastnet: a global data-driven high-resolution weather model using adaptive fourier neural operators. (2022). https://arxiv.org/abs/2202.11214.
[14]
Franz Poeschel et al. 2022. Transitioning from file-based hpc workflows to streaming data pipelines with openpmd and adios2. In Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation. Jeffrey Nichols, Arthur 'Barney' Maccabe, James Nutaro, Swaroop Pophale, Pravallika Devineni, Theresa Ahearn, and Becky Verastegui, (Eds.) Springer International Publishing, Cham, 99--118. isbn: 978-3-030-96498-6.
[15]
T. Rackow, X. Pedruzo-Bagazgoitia, S. Milinkski, I. Sandu, and et al. [n. d.] Multi-year simulations at kilometre scale with the integrated forecasting system coupled to fesom2.5/nemov3.4. Geoscientific Model Development. In preparation.
[16]
Simon Smart, Tiago Quintino, and Baudouin Raoult. 2019. A high-performance distributed object-store for exascale numerical weather prediction and climate. In Proceedings of the Platform for Advanced Scientific Computing Conference, 1--11.
[17]
[SW] The HDF Group, Hierarchical Data Format, version 5. url: https://github.com/HDFGroup/hdf5.
[18]
O. Treiber. 2023. ECMWF Bologna Data Centre's High-Performance-Computing Facility. Presented at the 20th ECMWF workshop on high performance computing in meteorology. (Oct. 2023). https://ecmwfevents.com/assets/presentations/hpcws2023-treiber1697006150.pdf.
[19]
UCAR/Unidata. 2023. Network common data form (netcdf). (2023). https://www.unidata.ucar.edu/software/netcdf/.
[20]
Michèle Weiland et al. 2019. An early evaluation of intel's optane dc persistent memory module and its impact on high-performance scientific applications. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '19) Article 76. Association for Computing Machinery, Denver, Colorado, 19 pages. isbn: 9781450362290.
[21]
X. Yepes-Arbós, G. van den Oord, M. C. Acosta, and G. D. Carver. 2022. Evaluation and optimisation of the i/o scalability for the next generation of earth system models: ifs cy43r3 and xios 2.0 integration as a case study. Geoscientific Model Development, 15, 2, 379--394.
Index Terms
- MultIO: A Framework for Message-Driven Data Routing For Weather and Climate Simulations
Recommendations
SARIMA Model: An Efficient Machine Learning Technique for Weather Forecasting
AbstractWeather forecasting is a critical tool for many different applications, from agriculture and transportation to disaster preparedness and response. While weather forecasts are not always perfect, they provide valuable information that can help ...
Development of a gridded climate data tool for the COordinated Regional climate Downscaling EXperiment data
The CDSL was developed to aid the processing of regional climate data.The CDSL supports parallel processing of gridded data for agricultural models.The CDSL has a functionality to prepare an ensemble set from individual data sets.Input files from ...
Deep learning-based effective fine-grained weather forecasting model
AbstractIt is well-known that numerical weather prediction (NWP) models require considerable computer power to solve complex mathematical equations to obtain a forecast based on current weather conditions. In this article, we propose a novel lightweight ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2024
296 pages
ISBN:9798400706394
DOI:10.1145/3659914
- Proceedings Chairs:
- Timothy Robinson,
- Michèle Woodtli,
- Program Chairs:
- Axel Huebl,
- Cristina Silvano
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- European Union
Conference
PASC '24
Sponsor:
Acceptance Rates
PASC '24 Paper Acceptance Rate 26 of 36 submissions, 72%;
Overall Acceptance Rate 109 of 221 submissions, 49%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 141Total Downloads
- Downloads (Last 12 months)141
- Downloads (Last 6 weeks)32
Reflects downloads up to 25 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in