research-article

The deep computing messaging framework: generalized scalable message passing on the blue gene/P supercomputer

Authors:

Gheorghe Almasi,

Philip Heidelberger,

Mark E. Giampapa,

Michael Blocksome,

Joseph Ratterman,

Charles J. ArcherAuthors Info & Claims

ICS '08: Proceedings of the 22nd annual international conference on Supercomputing

Pages 94 - 103

https://doi.org/10.1145/1375527.1375544

Published: 07 June 2008 Publication History

Abstract

We present the architecture of the Deep Computing Messaging Framework (DCMF), a message passing runtime designed for the Blue Gene/P machine and other HPC architectures. DCMF has been designed to easily support several programming paradigms such as the Message Passing Interface (MPI), Aggregate Remote Memory Copy Interface (ARMCI), Charm++, and others. This support is made possible as DCMF provides an application programming interface (API) with active messages and non-blocking collectives. DCMF is being open sourced and has a layered component based architecture with multiple levels of abstraction, allowing the members of the community to contribute new components to its design at the various layers. The DCMF runtime can be extended to other architectures through the development of architecture specific implementations of interface classes. The production DCMF runtime on Blue Gene/P takes advantage of the direct memory access (DMA) hardware to offload message passing work and achieve good overlap of computation and communication. We take advantage of the fact that the Blue Gene/P node is a symmetric multi-processor with four cache-coherent cores and use multi-threading to optimize the performance on the collective network. We also present a performance evaluation of the DCMF runtime on Blue Gene/P and show that it delivers performance close to hardware limits.

References

[1]

Open Fabrics Alliance. http://www.openfabrics.org

[2]

N. R. Adiga et al. Blue Gene/L torus interconnection network. IBM J. Res. Dev., 49:265--276, (2005)

Digital Library

[3]

G. Almasi et al. Design and implementation of message-passing services for the Blue Gene/L supercomputer. IBM J. Res. Dev., 49:393--406, (2005)

Digital Library

[4]

M. Banikazemi, R. Govindaraju, R. Blackmore, and D. K. Panda. MPI-LAPI: An efficient implementation of MPI for IBM RS/6000 SP systems. IEEE Transactions on Parallel and Distributed Systems, 12(10):1081--1093, 2001

Digital Library

[5]

C. Barton, C. Cascaval, S. Chatterjee, G. Almasi, Y. Zheng, M. Farreras, and J. Amaral. Shared memory programming for large scale machines. In Proceedings of ACM SIGPLAN Conference on Programming Language Design and Implementation, June 2006

Digital Library

[6]

DCMF. http://dcmf.anl-external.org/wiki, 2008

[7]

J. Dongarra, E. Strohmaier, H. Simon, and H. Meuer. www.top500.org, 2007. Date retrieved: 10 Jan 2008

[8]

M. P. I. Forum. MPI-2: Extensions to the message-passing interface, 1997. http://www.mpiforum.org/docs/mpi-20-html/mpi2-report.html

[9]

W. Gropp, E. Lusk, N. Doss, and A. Skjellum. Mpich: A high-performance, portable implementation of the mpi message passing interface standard. Parallel Computing, 22(6):789--828, September 1996

Digital Library

[10]

IBM Blue Gene Team. Overview of the Blue Gene/P project. IBM J. Res. Dev., 52(1/2), January (2008). http://www.research.ibm.com/journal/rd/521/team.html

Digital Library

[11]

L. V. Kale, M. Bhandarkar, N. Jagathesan, e S. Krishnan, and J. Yelon. Converse: An Interoperable Framework for Parallel Programming. In Proceedings of the 10th International Parallel Processing Symposium, pages 212--217, Honolulu, Hawaii, April 1996

Digital Library

[12]

L. V. Kale and S. Krishnan. Charm++: Parallel Programming with Message-Driven Objects. In G. V. Wilson and P. Lu, editors, Parallel Programming using C++, pages 175--213. MIT Press, 1996

[13]

S. Kumar, C. Huang, G. Almasi, and L. V. Kale Achieving strong scaling with NAMD on Blue Gene/L. In Proceedings of IEEE International Parallel and Distributed Processing Symposium 2006, April 2006

Digital Library

[14]

Myrinet Inc. Myrinet Express (MX), A High Performance Low Level Message Passing Interface for Myrinet, January 2006

[15]

J. Nieplocha and B. Carpenter. ARMCI: A portable remote memory copy library for distributed array libraries and compiler run-time systems. Lecture Notes in Computer Science, 1586, 1999.

Digital Library

[16]

F. Petrini, W. chun Feng, A. Hoisie, S. Coll, and E. Frachtenberg. The quadrics network: high-performance clustering technology. IEEE Micro, 22(1):46--57, 2002.

Digital Library

[17]

W. Gropp and E. Lusk. MPICH ADI Implementation Reference Manual, August 1995.

Cited By

Xu JPang ZDai Y(2024)Collective Communication Acceleration Architecture for Reduce Operation with Long Operands Based on Efficient Data Access and Reconfigurable On-Chip Buffer2024 IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA)10.1109/ISPA63168.2024.00174(1295-1302)Online publication date: 30-Oct-2024
https://doi.org/10.1109/ISPA63168.2024.00174
Protze JHermanns MMüller MNguyen VJaeger JSaillard ECarribault PBarthou D(2022)MPI detach — Towards automatic asynchronous local completionParallel Computing10.1016/j.parco.2021.102859109:COnline publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1016/j.parco.2021.102859
Thomadakis PTsolakis CChrisochoides N(2022)Multithreaded runtime framework for parallel and adaptive applicationsEngineering with Computers10.1007/s00366-022-01713-738:5(4675-4695)Online publication date: 31-Jul-2022
https://doi.org/10.1007/s00366-022-01713-7
Show More Cited By

Index Terms

The deep computing messaging framework: generalized scalable message passing on the blue gene/P supercomputer
1. Software and its engineering
  1. Software notations and tools
    1. Context specific languages
      1. Domain specific languages
  2. Software organization and properties
    1. Software system structures
      1. Software architectures

Recommendations

PAMI: A Parallel Active Message Interface for the Blue Gene/Q Supercomputer
IPDPS '12: Proceedings of the 2012 IEEE 26th International Parallel and Distributed Processing Symposium

The Blue Gene/Q machine is the next generation in the line of IBM massively parallel supercomputers, designed to scale to 262144 nodes and sixteen million threads. With each BG/Q node having 68 hardware threads, hybrid programming paradigms, which use ...
Architecture of the Component Collective Messaging Interface

Different programming paradigms utilize a variety of collective communication operations, often with different semantics. We present the component collective messaging interface (CCMI) that can support asynchronous non-blocking collectives and is ...
Implementation and evaluation of shared-memory communication and synchronization operations in MPICH2 using the Nemesis communication subsystem

This paper presents the implementation of MPICH2 over the Nemesis communication subsystem and the evaluation of its shared-memory performance. We describe design issues as well as some of the optimization techniques we employed. We conducted a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICS '08: Proceedings of the 22nd annual international conference on Supercomputing

June 2008

390 pages

ISBN:9781605581583

DOI:10.1145/1375527

General Chairs:
Theo Papatheodorou
University of Patras, Greece
,
Utpal Banerjee
Intel (retired), USA
,
Program Chairs:
Avi Mendelson
Intel, Israel
,
Kyle Gallivan
Florida State University, USA

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 June 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICS08

Sponsor:

ICS08: International Conference on Supercomputing

June 7 - 12, 2008

Island of Kos, Greece

Acceptance Rates

Overall Acceptance Rate 629 of 2,180 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

89
Total Citations
View Citations
717
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xu JPang ZDai Y(2024)Collective Communication Acceleration Architecture for Reduce Operation with Long Operands Based on Efficient Data Access and Reconfigurable On-Chip Buffer2024 IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA)10.1109/ISPA63168.2024.00174(1295-1302)Online publication date: 30-Oct-2024
https://doi.org/10.1109/ISPA63168.2024.00174
Protze JHermanns MMüller MNguyen VJaeger JSaillard ECarribault PBarthou D(2022)MPI detach — Towards automatic asynchronous local completionParallel Computing10.1016/j.parco.2021.102859109:COnline publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1016/j.parco.2021.102859
Thomadakis PTsolakis CChrisochoides N(2022)Multithreaded runtime framework for parallel and adaptive applicationsEngineering with Computers10.1007/s00366-022-01713-738:5(4675-4695)Online publication date: 31-Jul-2022
https://doi.org/10.1007/s00366-022-01713-7
Ouyang KSi MHori AChen ZBalaji P(2021)Daps: A Dynamic Asynchronous Progress Stealing Model for MPI Communication2021 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/Cluster48925.2021.00027(516-527)Online publication date: Sep-2021
https://doi.org/10.1109/Cluster48925.2021.00027
Protze JHermanns MDemiralp AMüller MKuhlen T(2020)MPI Detach - Asynchronous Local CompletionProceedings of the 27th European MPI Users' Group Meeting10.1145/3416315.3416323(71-80)Online publication date: 21-Sep-2020
https://dl.acm.org/doi/10.1145/3416315.3416323
Kang QRoss RLatham RLee SAgrawal AChoudhary ALiao W(2020)Improving All-to-Many Personalized Communication in Two-Phase I/OSC20: International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC41405.2020.00014(1-13)Online publication date: Nov-2020
https://doi.org/10.1109/SC41405.2020.00014
Ghazimirsaeed SMirsadeghi SAfsahi A(2019)An Efficient Collaborative Communication Mechanism for MPI Neighborhood Collectives2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS.2019.00087(781-792)Online publication date: May-2019
https://doi.org/10.1109/IPDPS.2019.00087
Schuchart JBouteiller ABosilca G(2019)Using MPI-3 RMA for Active Messages2019 IEEE/ACM Workshop on Exascale MPI (ExaMPI)10.1109/ExaMPI49596.2019.00011(47-56)Online publication date: Nov-2019
https://doi.org/10.1109/ExaMPI49596.2019.00011
Gooding TRosenburg BGiampapa MInglett TWisniewski R(2019)Blue Gene Line of LWKsOperating Systems for Supercomputers and High Performance Computing10.1007/978-981-13-6624-6_5(71-91)Online publication date: 16-Oct-2019
https://doi.org/10.1007/978-981-13-6624-6_5
Iskra KYoshii KBeckman P(2019)ZeptoOSOperating Systems for Supercomputers and High Performance Computing10.1007/978-981-13-6624-6_10(161-181)Online publication date: 16-Oct-2019
https://doi.org/10.1007/978-981-13-6624-6_10
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten