research-article

MPIX Stream: An Explicit Solution to Hybrid MPI+X Programming

Authors:

Ken Raffenetti,

Rajeev ThakurAuthors Info & Claims

EuroMPI/USA '22: Proceedings of the 29th European MPI Users' Group Meeting

Pages 1 - 10

https://doi.org/10.1145/3555819.3555820

Published: 14 September 2022 Publication History

Abstract

The hybrid MPI+X programming paradigm, where X refers to threads or GPUs, has gained prominence in the high-performance computing arena. This corresponds to a trend of system architectures growing more heterogeneous. The current MPI standard only specifies the compatibility levels between MPI and threading runtimes. No MPI concept or interface exists for applications to pass thread context or GPU stream context to MPI implementations explicitly. This lack has made performance optimization complicated in some cases and impossible in other cases. We propose a new concept in MPI, called MPIX stream, to represent the general serial execution context that exists in X runtimes. MPIX streams can be directly mapped to threads or GPU execution streams. Passing thread context into MPI allows implementations to precisely map the execution contexts to network endpoints. Passing GPU execution context into MPI allows implementations to directly operate on GPU streams, lowering the CPU/GPU synchronization cost.

References

[1]

Abdelhalim Amer, Huiwei Lu, Yanjie Wei, Pavan Balaji, and Satoshi Matsuoka. 2015. MPI+ threads: Runtime contention and remedies. ACM SIGPLAN Notices 50, 8 (2015), 239–248.

Digital Library

[2]

James Dinan, Ryan E Grant, Pavan Balaji, David Goodell, Douglas Miller, Marc Snir, and Rajeev Thakur. 2014. Enabling communication concurrency through flexible MPI endpoints. The International Journal of High Performance Computing Applications 28, 4(2014), 390–405.

Digital Library

[3]

Ryan E Grant, Matthew GF Dosanjh, Michael J Levenhagen, Ron Brightwell, and Anthony Skjellum. 2019. Finepoints: Partitioned multithreaded MPI communication. In International Conference on High Performance Computing. Springer, 330–350.

[4]

Nathan Hjelm, Howard Pritchard, Samuel K. Gutiérrez, Daniel J. Holmes, Ralph Castain, and Anthony Skjellum. 2019. MPI Sessions: Evaluation of an Implementation in Open MPI. In 2019 IEEE International Conference on Cluster Computing (CLUSTER). 1–11.

[5]

Argonne National Laboratory. 2022. MPICH. https://www.mpich.org/.

[6]

Message Passing Interface Forum. 2021. MPI: A Message-Passing Interface Standard, Version 4.0. https://www.mpi-forum.org/docs/.

[7]

Naveen Namashivayam, Krishna Kandalla, Trey White, Nick Radcliffe, Larry Kaplan, and Mark Pagel. 2022. Exploring GPU Stream-Aware Message Passing using Triggered Operations. https://doi.org/10.48550/ARXIV.2208.04817

[8]

NVIDIA. 2022. NCCL. https://developer.nvidia.com/nccl.

[9]

Open Fabric Alliance. 2022. libfabric. https://github.com/ofiwg/libfabric.

[10]

Open UCX. 2022. UCX. https://github.com/openucx/ucx.

[11]

Thananon Patinyasakdikul, David Eberius, George Bosilca, and Nathan Hjelm. 2019. Give MPI threading a fair chance: A study of multithreaded MPI designs. In 2019 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 1–11.

[12]

The Open MPI Team. 2022. Open MPI. https://www.open-mpi.org/.

[13]

The Ohio State University. 2022. MVAPICH. https://mvapich.cse.ohio-state.edu/.

[14]

Hengjie Wang and Aparna Chandramowlishwaran. 2019. Multi-criteria partitioning of multi-block structured grids. In Proceedings of the ACM International Conference on Supercomputing. 261–271.

Digital Library

[15]

Hao Wang, Sreeram Potluri, Devendar Bureddy, Carlos Rosales, and Dhabaleswar K. Panda. 2014. GPU-Aware MPI on RDMA-Enabled Clusters: Design, Implementation and Evaluation. IEEE Transactions on Parallel and Distributed Systems 25, 10 (2014), 2595–2605. https://doi.org/10.1109/TPDS.2013.222

[16]

Rohit Zambre, Aparna Chandramowliswharan, and Pavan Balaji. 2020. How I learned to stop worrying about user-visible endpoints and love MPI. In Proceedings of the 34th ACM International Conference on Supercomputing. 1–13.

Digital Library

[17]

Rohit Zambre, Damodar Sahasrabudhe, Hui Zhou, Martin Berzins, Aparna Chandramowlishwaran, and Pavan Balaji. 2021. Logically Parallel Communication for Fast MPI+ Threads Applications. IEEE Transactions on Parallel and Distributed Systems 32, 12 (2021), 3038–3052.

Digital Library

Cited By

Zhou HRaffenetti KGuo YGillis TLatham RThakur R(2024)Designing and prototyping extensions to the Message Passing Interface in MPICHThe International Journal of High Performance Computing Applications10.1177/10943420241263544Online publication date: 19-Aug-2024
https://doi.org/10.1177/10943420241263544
Huang YFan XYan SWeng C(2024)Neos: A NVMe-GPUs Direct Vector Service Buffer in User Space2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00289(3767-3781)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00289
Schuchart JGabriel E(2024)Stream Support in MPI Without the ChurnRecent Advances in the Message Passing Interface10.1007/978-3-031-73370-3_4(56-72)Online publication date: 25-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73370-3_4
Show More Cited By

Recommendations

Hybrid MPI: a case study on the Xeon Phi platform
ROSS '14: Proceedings of the 4th International Workshop on Runtime and Operating Systems for Supercomputers

New many-core architectures such as Intel Xeon Phi offer applications significantly higher power efficiency than conventional multi-core processors. However, while this processor's compute and communication performance is an excellent match for MPI ...
MPI+ULT: Overlapping Communication and Computation with User-Level Threads
HPCC-CSS-ICESS '15: Proceedings of the 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security, and 2015 IEEE 12th International Conf on Embedded Software and Systems

As the core density of future processors keeps increasing, MPI+Threads is becoming a promising programming model for large scale SMP clusters. Generally speaking, hybrid MPI+Threads runtime can largely improve intra-node parallelism and data sharing on ...
Understanding GPU Triggering APIs for MPI+X Communication
Recent Advances in the Message Passing Interface
Abstract
GPU-enhanced architectures are now dominant in HPC systems, but message-passing communication involving GPUs with MPI has proven to be both complex and expensive, motivating new approaches that lower such costs. We compare and contrast stream/...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

EuroMPI/USA '22: Proceedings of the 29th European MPI Users' Group Meeting

September 2022

78 pages

ISBN:9781450397995

DOI:10.1145/3555819

Copyright © 2022 ACM.

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 September 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

EuroMPI/USA'22

EuroMPI/USA'22: 29th European MPI Users' Group Meeting

September 26 - 28, 2022

TN, Chattanooga, USA

Acceptance Rates

Overall Acceptance Rate 66 of 139 submissions, 47%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
152
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)2

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhou HRaffenetti KGuo YGillis TLatham RThakur R(2024)Designing and prototyping extensions to the Message Passing Interface in MPICHThe International Journal of High Performance Computing Applications10.1177/10943420241263544Online publication date: 19-Aug-2024
https://doi.org/10.1177/10943420241263544
Huang YFan XYan SWeng C(2024)Neos: A NVMe-GPUs Direct Vector Service Buffer in User Space2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00289(3767-3781)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00289
Schuchart JGabriel E(2024)Stream Support in MPI Without the ChurnRecent Advances in the Message Passing Interface10.1007/978-3-031-73370-3_4(56-72)Online publication date: 25-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73370-3_4
Bridges PSkjellum ASuggs ESchafer DBangalore P(2024)Understanding GPU Triggering APIs for MPI+X CommunicationRecent Advances in the Message Passing Interface10.1007/978-3-031-73370-3_3(39-55)Online publication date: 25-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73370-3_3
Yan JKaiser HSnir M(2023)Design and Analysis of the Network Software Stack of an Asynchronous Many-task System -- The LCI parcelport of HPXProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624598(1151-1161)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624598
Zhou HRaffenetti KZhang JGuo YThakur R(2023)Frustrated With MPI+Threads? Try MPIxThreads!Proceedings of the 30th European MPI Users' Group Meeting10.1145/3615318.3615320(1-10)Online publication date: 11-Sep-2023
https://dl.acm.org/doi/10.1145/3615318.3615320
Gillis TRaffenetti KZhou HGuo YThakur R(2023)Quantifying the Performance Benefits of Partitioned Communication in MPIProceedings of the 52nd International Conference on Parallel Processing10.1145/3605573.3605599(285-294)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3605573.3605599
Anthony QAwan ARasley JHe YShafi AAbduljabbar MSubramoni HPanda D(2023)MCR-DL: Mix-and-Match Communication Runtime for Deep Learning2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS54959.2023.00103(996-1006)Online publication date: May-2023
https://doi.org/10.1109/IPDPS54959.2023.00103

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents