article

Overview of the Blue Gene/L system architecture

Authors:

M. A. Blumrich,

M. E. Giampapa,

P. Heidelberger,

B. D. Steinmacher-Burow,

P. VranasAuthors Info & Claims

IBM Journal of Research and Development, Volume 49, Issue 2

Pages 195 - 212

https://doi.org/10.1147/rd.492.0195

Published: 01 March 2005 Publication History

Abstract

The Blue Gene®/L computer is a massively parallel supercomputer based on IBM system-on-a-chip technology. It is designed to scale to 65,536 dual-processor nodes, with a peak performance of 360 teraflops. This paper describes the project objectives and provides an overview of the system architecture that resulted. We discuss our application-based approach and rationale for a low-power, highly integrated design. The key architectural features of Blue Gene/L are introduced in this paper: the link chip component and five Blue Gene/L networks, the PowerPC® 440 core and floating-point enhancements, the on-chip and off-chip distributed memory system, the node- and system-level design for high reliability, and the comprehensive approach to fault isolation.

References

[1]

P. A. Boyle, C. Jung, and T. Wettig, "The QCDOC Supercomputer: Hardware, Software, and Performance," Proceedings of the Conference for Computing in High Energy and Nuclear Physics (CHEP03), 2003; see paper at http:// xxx.lanl.gov/PS_cache/hep-lat/pdf/0306/0306023.pdf.

[2]

P. A. Boyle, D. Chen, N. H. Christ, M. A. Clark, S. D. Cohen, C. Cristian, Z. Dong, A. Gara, B. Joó, C. Jung, C. Kim, L. A. Levkova, X. Liao, G. Liu, R. D. Mawhinney, S. Ohta, K. Petrov, T. Wettig, and A. Yamaguchi, "Overview of the QCDSP and QCDOC Computers," IBM J. Res. & Dev. 49, No. 2/3, 351-365 (2005, this issue).

Digital Library

[3]

"Electrical Energy," The New Book of Popular Science, Grolier Incorporated, Danbury, CT, 2000.

[4]

X. Martorell, N. Smeds, R. Walkup, J. R. Brunheroto, G. Almási, J. A. Gunnels, L. DeRose, J. Labarta, F. Escalé, J. Giménez, H. Servat, and J. E. Moreira, "Blue Gene/L Performance Tools," IBM J. Res. & Dev. 49, No. 2/3, 407-424 (2005, this issue).

Digital Library

[5]

Message Passing Interface Forum, "MPI: A Message-Passing Interface Standard," University of Tennessee, 1995; see http:// www.mpi-forum.org/docs/mpi-11-html/mpi-report.html.

[6]

G. Almási, C. Archer, J. G. Castaños, J. A. Gunnels, C. C. Erway, P. Heidelberger, X. Martorell, J. E. Moreira, K. Pinnow, J. Ratterman, B. D. Steinmacher-Burow, W. Gropp, and B. Toonen, "Design and Implementation of Message-Passing Services for the Blue Gene/L Supercomputer," IBM J. Res. & Dev. 49, No. 2/3, 393-406 (2005, this issue).

Digital Library

[7]

A. A. Bright, R. A. Haring, M. B. Dombrowa, M. Ohmacht, D. Hoenicke, S. Singh, J. A. Marcella, R. F. Lembach, S. M. Douskey, M. R. Ellavsky, C. G. Zoellin, and A. Gara, "Blue Gene/L Compute Chip: Synthesis, Timing, and Physical Design," IBM J. Res. & Dev. 49, No. 2/3, 277-287 (2005, this issue).

Digital Library

[8]

M. E. Wazlowski, N. R. Adiga, D. K. Beece, R. Bellofatto, M. A. Blumrich, D. Chen, M. B. Dombrowa, A. Gara, M. E. Giampapa, R. A. Haring, P. Heidelberger, D. Hoenicke, B. J. Nathanson, M. Ohmacht, R. Sharrar, S. Singh, B. D. Steinmacher-Burow, R. B. Tremaine, M. Tsao, A. R. Umamaheshwaran, and P. Vranas, "Verification Strategy for the Blue Gene/L Chip," IBM J. Res. & Dev. 49, No. 2/3, 303-318 (2005, this issue).

Digital Library

[9]

M. E. Giampapa, R. Bellofatto, M. A. Blumrich, D. Chen, M. B. Dombrowa, A. Gara, R. A. Haring, P. Heidelberger, D. Hoenicke, G. V. Kopcsay, B. J. Nathanson, B. D. Steinmacher-Burow, M. Ohmacht, V. Salapura, and P. Vranas, "Blue Gene/L Advanced Diagnostics Environment," IBM J. Res. & Dev. 49, No. 2/3, 319-331 (2005, this issue).

Digital Library

[10]

R. S. Germain, Y. Zhestkov, M. Eleftheriou, A. Rayshubskiy, F. Suits, T. J. C. Ward, and B. G. Fitch, "Early Performance Data on the Blue Matter Molecular Simulation Framework," IBM J. Res. & Dev. 49, No. 2/3, 447-455 (2005, this issue).

Digital Library

[11]

M. Eleftheriou, B. G. Fitch, A. Rayshubskiy, T. J. C. Ward, and R. S. Germain, "Scalable Framework for 3D FFTs on the Blue Gene/L Supercomputer: Implementation and Early Performance Measurements," IBM J. Res. & Dev. 49, No. 2/3, 457-464 (2005, this issue).

Digital Library

[12]

F. Suits, M. C. Pitman, J. W. Pitera, W. C. Swope, and R. S. Germain, "Overview of Molecular Dynamics Techniques and Early Scientific Results from the Blue Gene Project," IBM J. Res. & Dev. 49, No. 2/3, 475-487 (2005, this issue).

Digital Library

[13]

G. Almási, S. Chatterjee, A. Gara, J. Gunnels, M. Gupta, A. Henning, J. E. Moreira, B. Walkup, A. Curioni, C. Archer, L. Bachega, B. Chan, B. Curtis, M. Brodowicz, S. Brunett, E. Upchurch, G. Chukkapalli, R. Harkness, and W. Pfeiffer, "Unlocking the Performance of the BlueGene/L Supercomputer," Proceedings of SC'04, 2004; see paper at http://www.sc-conference.org/sc2004/schedule/pdfs/pap220.pdf.

[14]

See http://www-306.ibm.com/chips/techlib/techlib.nsf/techdocs/ 05D0405273F1C1BD87256D6D0063CFB9.

[15]

S. S. Iyer, J. E. Barth, Jr., P. C. Parries, J. P. Norum, J. P. Rice, L. R. Logan, and D. Hoyniak, "Embedded DRAM: Technology Platform for the Blue Gene/L Chip," IBM J. Res. & Dev. 49, No. 2/3, 333-350 (2005, this issue).

Digital Library

[16]

P. Coteus, H. R. Bickford, T. M. Cipolla, P. G. Crumley, A. Gara, S. A. Hall, G. V. Kopcsay, A. P. Lanzetta, L. S. Mok, R. Rand, R. Swetz, T. Takken, P. La Rocca, C. Marroquin, P. R. Germann, and M. J. Jeanson, "Packaging the Blue Gene/L Supercomputer," IBM J. Res. & Dev. 49, No. 2/3, 213-248 (2005, this issue).

Digital Library

[17]

Y. Aridor, T. Domany, O. Goldshmidt, J. E. Moreira, and E. Shmueli, "Resource Allocation and Utilization in the Blue Gene/L Supercomputer," IBM J. Res. & Dev. 49, No. 2/3, 425-436 (2005, this issue).

Digital Library

[18]

M. Ohmacht, R. A. Bergamaschi, S. Bhattacharya, A. Gara, M. E. Giampapa, B. Gopalsamy, R. A. Haring, D. Hoenicke, D. J. Krolak, J. A. Marcella, B. J. Nathanson, V. Salapura, and M. E. Wazlowski, "Blue Gene/L Compute Chip: Memory and Ethernet Subsystem," IBM J. Res. & Dev. 49, No. 2/3, 255-264 (2005, this issue).

Digital Library

[19]

N. R. Adiga, M. A. Blumrich, D. Chen, P. Coteus, A. Gara, M. E. Giampapa, P. Heidelberger, S. Singh, B. D. Steinmacher-Burow, T. Takken, M. Tsao, and P. Vranas, "Blue Gene/L Torus Interconnection Network," IBM J. Res. & Dev. 49, No. 2/3, 265-276 (2005, this issue).

Digital Library

[20]

R. A. Haring, R. Bellofatto, A. A. Bright, P. G. Crumley, M. B. Dombrowa, S. M. Douskey, M. R. Ellavsky, B. Gopalsamy, D. Hoenicke, T. A. Liebsch, J. A. Marcella, and M. Ohmacht, "Blue Gene/L Compute Chip: Control, Test, and Bring-Up Infrastructure," IBM J. Res. & Dev. 49, No. 2/3, 289-301 (2005, this issue).

Digital Library

[21]

G. Bhanot, A. Gara, P. Heidelberger, E. Lawless, J. C. Sexton, and R. Walkup, "Optimizing Task Layout on the Blue Gene/L Supercomputer," IBM J. Res. & Dev. 49, No. 2/3, 489-500 (2005, this issue).

Digital Library

[22]

J. E. Moreira, G. Almási, C. Archer, R. Bellofatto, P. Bergner, J. R. Brunheroto, M. Brutman, J. G. Castaños, P. G. Crumley, M. Gupta, T. Inglett, D. Lieber, D. Limpert, P. McCarthy, M. Megerian, M. Mendell, M. Mundy, D. Reed, R. K. Sahoo, A. Sanomiya, R. Shok, B. Smith, and G. G. Stewart, "Blue Gene/L Programming and Operating Environment," IBM J. Res. & Dev. 49, No. 2/3, 367-376 (2005, this issue).

Digital Library

[23]

S. Chatterjee, L. R. Bachega, P. Bergner, K. A. Dockser, J. A. Gunnels, M. Gupta, F. G. Gustavson, C. A. Lapkowski, G. K. Liu, M. Mendell, R. Nair, C. D. Wait, T. J. C. Ward, and P. Wu, "Design and Exploitation of a High-Performance SIMD Floating-Point Unit for Blue Gene/L," IBM J. Res. & Dev. 49, No. 2/3, 377-391 (2005, this issue).

Digital Library

[24]

C. D. Wait, "IBM PowerPC 440 FPU with Complex-Arithmetic Extensions," IBM J. Res. & Dev. 49, No. 2/3, 249-254 (2005, this issue).

Digital Library

[25]

J. Lorenz, S. Kral, F. Franchetti, and C. W. Ueberhuber, "Vectorization Techniques for the Blue Gene/L Double FPU," IBM J. Res. & Dev. 49, No. 2/3, 437-446 (2005, this issue).

Digital Library

[26]

R. F. Enenkel, B. G. Fitch, R. S. Germain, F. G. Gustavson, A. Martin, M. Mendell, J. W. Pitera, M. C. Pitman, A. Rayshubskiy, F. Suits, W. C. Swope, and T. J. C. Ward, "Custom Math Functions for Molecular Dynamics," IBM J. Res. & Dev. 49, No. 2/3, 465-474 (2005, this issue).

Digital Library

Cited By

Collom GLi RBienz A(2023)Optimizing Irregular Communication with Neighborhood Collectives and Locality-Aware ParallelismProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624111(427-437)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624111
Klenk BJiang NThorson GDennison LMartínez JDuato JEeckhout L(2020)An in-network architecture for accelerating shared-memory multiprocessor collectivesProceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture10.1109/ISCA45697.2020.00085(996-1009)Online publication date: 30-May-2020
https://dl.acm.org/doi/10.1109/ISCA45697.2020.00085
Navaridas JLant JPascual JLuján MGoodacre J(2019)Design Exploration of Multi-tier Interconnection Networks for Exascale SystemsProceedings of the 48th International Conference on Parallel Processing10.1145/3337821.3337903(1-10)Online publication date: 5-Aug-2019
https://dl.acm.org/doi/10.1145/3337821.3337903
Show More Cited By

Overview of the Blue Gene/L system architecture

Recommendations

Blue Gene: A Massively Parallel System
ICCS '01: Proceedings of the International Conference on Computational Sciences-Part I

Blue Gene is a massively parallel system being developed at the IBM T. J. Watson Research Center. With its 4 million-way parallelism and 1 Petaflop peak performance, Blue Gene is a unique environment for research in parallel processing. Full ...
IBM System Blue Gene Solution: Blue Gene/P Application Development
IBM System Blue Gene Solution: Blue Gene/P Application Development

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IBM Journal of Research and Development

IBM Journal of Research and Development Volume 49, Issue 2

March 2005

302 pages

ISSN:0018-8646

Issue’s Table of Contents

Publisher

IBM Corp.

United States

Publication History

Published: 01 March 2005

Accepted: 03 December 2004

Received: 29 October 2004

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

99
Total Citations
View Citations
12
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Collom GLi RBienz A(2023)Optimizing Irregular Communication with Neighborhood Collectives and Locality-Aware ParallelismProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624111(427-437)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624111
Klenk BJiang NThorson GDennison LMartínez JDuato JEeckhout L(2020)An in-network architecture for accelerating shared-memory multiprocessor collectivesProceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture10.1109/ISCA45697.2020.00085(996-1009)Online publication date: 30-May-2020
https://dl.acm.org/doi/10.1109/ISCA45697.2020.00085
Navaridas JLant JPascual JLuján MGoodacre J(2019)Design Exploration of Multi-tier Interconnection Networks for Exascale SystemsProceedings of the 48th International Conference on Parallel Processing10.1145/3337821.3337903(1-10)Online publication date: 5-Aug-2019
https://dl.acm.org/doi/10.1145/3337821.3337903
Liu CHale KHoefler TIskra K(2019)Towards a Practical Ecosystem of Specialized OS KernelsProceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers10.1145/3322789.3328742(3-9)Online publication date: 17-Jun-2019
https://dl.acm.org/doi/10.1145/3322789.3328742
Fernando VFranques AAbadal SMisailovic STorrellas JBahar IHerlihy MWitchel ELebeck A(2019)ReplicaProceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3297858.3304033(849-863)Online publication date: 4-Apr-2019
https://dl.acm.org/doi/10.1145/3297858.3304033
Hu SZhu YCheng PGuo CTan KPadhye JChen K(2019)TaggerIEEE/ACM Transactions on Networking10.1109/TNET.2019.290287527:2(889-902)Online publication date: 1-Apr-2019
https://dl.acm.org/doi/10.1109/TNET.2019.2902875
Namassivaya NPal SRatnam D(2019)Modelling of FPGA-Particle Swarm Optimized GNSS Receiver for Satellite ApplicationsWireless Personal Communications: An International Journal10.1007/s11277-019-06193-5106:2(879-895)Online publication date: 1-May-2019
https://dl.acm.org/doi/10.1007/s11277-019-06193-5
Graham RBloch GBureddy DShainer GSmith B(2017)Towards A Data Centric System ArchitectureSupercomputing Frontiers and Innovations: an International Journal10.14529/jsfi1704014:4(4-16)Online publication date: 15-Dec-2017
https://dl.acm.org/doi/10.14529/jsfi170401
Scullin WScovel A(2017)Lessons from the IBM Blue Gene Series of SupercomputersProceedings of the HPC Systems Professionals Workshop10.1145/3155105.3155111(1-7)Online publication date: 12-Nov-2017
https://dl.acm.org/doi/10.1145/3155105.3155111
Hu SZhu YCheng PGuo CTan KPadhye JChen K(2017)TaggerProceedings of the 13th International Conference on emerging Networking EXperiments and Technologies10.1145/3143361.3143382(451-463)Online publication date: 28-Nov-2017
https://dl.acm.org/doi/10.1145/3143361.3143382
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents