research-article

The universe at extreme scale: multi-petaflop sky simulation on the BG/Q

Authors:

Vitali Morozov,

Katrin Heitmann,

Kalyan Kumaran,

Patricia Fasel,

Nicholas Frontiere,

Zarija LukićAuthors Info & Claims

SC '12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

Article No.: 4, Pages 1 - 11

Published: 10 November 2012 Publication History

Abstract

Remarkable observational advances have established a compelling cross-validated model of the Universe. Yet, two key pillars of this model -- dark matter and dark energy -- remain mysterious. Next-generation sky surveys will map billions of galaxies to explore the physics of the 'Dark Universe'. Science requirements for these surveys demand simulations at extreme scales; these will be delivered by the HACC (Hybrid/Hardware Accelerated Cosmology Code) framework. HACC's novel algorithmic structure allows tuning across diverse architectures, including accelerated and multi-core systems.

On the IBM BG/Q, HACC attains unprecedented scalable performance -- currently 6.23 PFlops at 62% of peak and 92% parallel efficiency on 786,432 cores (48 racks) -- at extreme problem sizes with up to almost two trillion particles, larger than any cosmological simulation yet performed. HACC simulations at these scales will for the first time enable tracking individual galaxies over the entire volume of a cosmological survey.

References

[1]

Albrecht, A., G. Bernstein, R. Cahn, W. L. Freedman, J. Hewitt, W. Hu, J. Huth, M. Kamionkowski, E. W. Kolb, L. Knox, J. C. Mather, S. Staggs, N. B. Suntzeff, Dark Energy Task Force Report, arXiv:astro-ph/0609591v1.

[2]

Alimi, J.-M., V. Bouillet, Y. Rasera, V. Reverdy, P.-S. Corasaniti, I. Balmes, S. Requena, X. Delaruelle, and J.-N. Richet, arXiv:1206.2838v1 {astro-ph.CO}.

[3]

Angulo, R. E., V. Springel, S. D. M. White, A. Jenkins, C. M. Baugh, and C. S. Frenk, arXiv:1203.3216v1 {astro-ph.CO}.

[4]

Bhattacharya, S., S. Habib, and K. Heitmann, arXiv:1112.5479v1 {astro-ph.CO}.

[5]

Chen, D., N. A. Eisley, P. Heidelberger, R. M. Senger, Y. Sugawara, S. Kumar, V. Salapura, D. L. Satterfield, B. Steinmacher-Burowin, and J. J. Parker, in SC11, Proceedings of the 2011 Int. Conf. High Performance Computing Networking Storage and Analysis (2011).

[6]

Clowe, D., S. W. Randall, and M. Markevitch, Nuc. Phys. Proc. Supp. 173, 28 (2007).

[7]

DOE ASCR and DOE HEP Joint Extreme Scale Computing Report (2009) -- "Challenges for Understanding the Quantum Universe and the Role of Computing at the Extreme Scale", http://extremecomputing.labworks.org/highenergyphysics/index.stm

[8]

For a recent review of cosmological simulation methods, see Dolag, K., S. Borgani, S. Schindler, A. Diaferio, and A. M. Bykov, Space Sci. Rev. 134, 229 (2008).

[9]

Evrard, A. E., T. MacFarland, H. M. P. Couchman, J. M. Colberg, N. Yoshida, S. D. M. White, A. Jenkins, C. S. Frenk, F. R. Pearce, G. Efstathiou, J. A. Peacock, and P. A. Thomas, Astrophys. J. 573, 7 (2002).

[10]

Frieman, J. A., M. S. Turner, and D. Huterer, Ann. Rev. Astron. & Astrophys. 46, 385 (2008).

[11]

Gafton, E. and S. Rosswog, Mon. Not. R. Astron. Soc., 418, 770 (2011).

[12]

Habib, S., unpublished.

[13]

Habib, S., A. Pope, Z. Lukić, D. Daniel, P. Fasel, N. Desai, K. Heitmann, C.-H. Hsu, L. Ankeny, G. Mark, S. Bhattacharya, and J. Ahrens, J. Phys. Conf. Ser. 180, 012019, (2009).

[14]

Haiman, Z., J. J. Mohr, and G. P. Holder, Astrophys. J. 553, 545 (2000).

[15]

Hamming, R. W., Digital Filters (Dover, 1998).

[16]

Haring, R. A., M. Ohmacht, T. W. Fox, M. K. Gschwind, P. A. Boyle, N. H. Christ, C. Kim, D. L. Satterfield, K. Sugavanam, P. W. Coteus, P. Heidelberger, M. A. Blumrich, R. W. Wisniewski, A. Gara, and G. L. Chiu, IEEE Micro 32, 48 (2012).

Digital Library

[17]

Hearin, A., A. Zentner, and Z. Ma, arXiv:1111.0052v2 {astro-ph.CO}.

[18]

Heitmann, K., P. M. Ricker, M. S. Warren and S. Habib, Astrophys. J. Supp. 160, 28 (2005).

[19]

Heitmann, K., D. Higdon, M. White, S. Habib, B. J. Williams, E. Lawrence, and C. Wagner, Astrophys. J. 705, 156 (2009).

[20]

Hockney, R. W. and J. W. Eastwood, Computer Simulation Using Particles (New York: Adam Hilger, 1988).

Digital Library

[21]

IBM Blue Gene/Q Application Development Redbook; available at: www.redbooks.ibm.com/redpieces/pdfs/sg247948.pdf

[22]

Kim, J., C. Park, G. Rosi, S. M. Lee, and J. R. Gott III, J. Korean Astron. Soc. 44, 217 (2011).

[23]

LSST website: www.lsst.org

[24]

Menanteau, F., J. P. Hughes, C. Sifon, M. Hilton, J. Gonzalez, L. Infante, L. F. Barrientos, A. J. Baker, J. R. Bond, S. Das, M. J. Devlin, J. Dunkley, A. Hajian, A. D. Hincks, A. Kosowsky, D. Marsden, T. A. Marriage, K. Moodley, M. D. Niemack, M. R. Nolta, L. A. Page, E. D. Reese, N. Sehgal, J. Sievers, D. N. Spergel, S. T. Staggs, E. Wollack, Astrophys. J. 748, 1 (2012).

[25]

Mortonson, M. J., W. Hu, and D. Huterer, Phys. Rev. D 83, 023015 (2011).

[26]

Myers, S. T., arXiv:0904.2593v1 {astro-ph.CO}.

[27]

Peebles, P. J. E., The Large-Scale Structure of the Universe (Princeton University Press, 1980).

[28]

Pfalzner, S., and P. Gibbon, Many-Body Tree Methods in Physics (Cambridge University Press, 1996); see also Barnes, J., and P. Hut, Nature 324, 446 (1986); Warren, M. S. and J. K. Salmon, Technical Paper, Supercomputing 1993.

[29]

Pope, A., S. Habib, Z. Lukic, D. Daniel, P. Fasel, N. Desai, and K. Heitmann, Computing in Science and Engineering 12, 17, 2010.

Digital Library

[30]

Power, C., J. F. Navarro, A. Jenkins, C. S. Frenk, S. D. M. White, V. Springel, J. Stadel, and T. Quinn, Mon. Not. R. Astron. Soc. 338, 14 (2003).

[31]

Qiang, J., R. D. Ryne, S. Habib, and V. Decyk, J. Comp. Phys. 163, 434 (2000).

Digital Library

[32]

Shandarin, S. F., and Ya.B. Zeldovich, Rev. Mod. Phys. 61, 185 (1989). See also, Shandarin, S., S. Habib, and K. Heitmann, Phys. Rev. D 85, 083005 (2012).

[33]

SDSS website: www.sdss.org

[34]

Springel, V., S. D. M. White, A. Jenkins, C. S. Fenk, N. Yoshida, L. Gao, J. Navarro, R. Thacker, D. Croton, J. Helly et al., Nature 435, 629 (2005).

[35]

Vikhlinin, A., A. V. Kravtsov, R. A. Burenin, H. Ebeling, W. R. Forman, A. Hornstrup, C. Jones and S. S. Murray et al., Astrophys. J. 692, 1060 (2009).

[36]

Weinberg, D. H., M. J. Mortonson, D. J. Eisenstein, C. Hirata, A. G. Riess and E. Rozo, arXiv:1201.2434 {astro-ph.CO}.

[37]

White, M. Astron. and Astrophys. 367, 27 (2001).

[38]

White, M., A. Pope, J. Carlson, K. Heitmann, S. Habib, P. Fasel, D. Daniel, and Z. Lukić, Astrophys. J. 713, 383 (2010).

[39]

Wittman, D. M., J. A. Tyson, I. P. Dell'Antonio, A. C. Becker, V. E. Margoniner, J. Cohen, D. Norman, D. Loomba, G. Squires, G. Wilson, C. Stubbs, J. Hennawi, D. Spergel, P. Boeshaar, A. Clocchiatti, M. Hamuy, G. Bernstein, A. Gonzalez, P. Guhathakurta, W. Hu, U. Seljak, and D. Zaritsky, arXiv:astro-ph/0210118.

Cited By

Aupy GGainaru AFèvre V(2019)I/O Scheduling Strategy for Periodic ApplicationsACM Transactions on Parallel Computing10.1145/33385106:2(1-26)Online publication date: 23-Jul-2019
https://dl.acm.org/doi/10.1145/3338510
Lockwood GSnyder SWang TByna SCarns PWright N(2018)A year in the life of a parallel file systemProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.5555/3291656.3291755(1-13)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.5555/3291656.3291755
Malakar PMunson TKnight CVishwanath VPapka M(2018)Topology-aware space-shared co-analysis of large-scale molecular dynamics simulationsProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.5555/3291656.3291688(1-15)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.5555/3291656.3291688
Show More Cited By

The universe at extreme scale: multi-petaflop sky simulation on the BG/Q

Recommendations

The Universe at extreme scale: Multi-petaflop sky simulation on the BG/Q
SC '12: Proceedings of the 2012 International Conference for High Performance Computing, Networking, Storage and Analysis

Remarkable observational advances have established a compelling cross-validated model of the Universe. Yet, two key pillars of this model--dark matter and dark energy--remain mysterious. Next-generation sky surveys will map billions of galaxies to ...
Titan Takes on the Universe: Large-Scale Cosmology Simulation of Universe Mines for Halos Where Galaxies are Born
A multi-institutional team led by researchers at the US Department of Energy's (DoE) Argonne National Laboratory has achieved high scalability and sustained performance on three of the most powerful systems in the world: IBM Blue Gene/Q supercomputers ...
GPU computing and the road to extreme-scale parallel systems
IISWC '11: Proceedings of the 2011 IEEE International Symposium on Workload Characterization

While Moore's Law has continued to provide smaller semiconductor devices, the effective end of uniprocessor performance scaling has (finally) instigated mainstream computing to adopt parallel hardware and software. Based on their derivation from high-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

November 2012

1161 pages

ISBN:9781467308045

General Chair:
Jeffrey K. Hollingsworth
University of Maryland

Sponsors

Publisher

IEEE Computer Society Press

Washington, DC, United States

Publication History

Published: 10 November 2012

Check for updates

Qualifiers

Research-article

Conference

SC '12

Sponsor:

SIGHPC
SIGARCH
IEEE-CS

SC '12: International Conference for High Performance Computing, Networking, Storage and Analysis

November 10 - 16, 2012

Utah, Salt Lake City

Acceptance Rates

SC '12 Paper Acceptance Rate 100 of 461 submissions, 22%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
228
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Aupy GGainaru AFèvre V(2019)I/O Scheduling Strategy for Periodic ApplicationsACM Transactions on Parallel Computing10.1145/33385106:2(1-26)Online publication date: 23-Jul-2019
https://dl.acm.org/doi/10.1145/3338510
Lockwood GSnyder SWang TByna SCarns PWright N(2018)A year in the life of a parallel file systemProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.5555/3291656.3291755(1-13)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.5555/3291656.3291755
Malakar PMunson TKnight CVishwanath VPapka M(2018)Topology-aware space-shared co-analysis of large-scale molecular dynamics simulationsProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.5555/3291656.3291688(1-15)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.5555/3291656.3291688
Lockwood GSnyder SWang TByna SCarns PWright N(2018)A year in the life of a parallel file systemProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC.2018.00077(1-13)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.1109/SC.2018.00077
Malakar PMunson TKnight CVishwanath VPapka M(2018)Topology-aware space-shared co-analysis of large-scale molecular dynamics simulationsProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC.2018.00027(1-15)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.1109/SC.2018.00027
Matri PPérez MCostan AAntoniu GEl-Araby EEl-Ghazawi TPanda D(2018)TýrFSProceedings of the 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing10.1109/CCGRID.2018.00072(452-461)Online publication date: 1-May-2018
https://dl.acm.org/doi/10.1109/CCGRID.2018.00072
Snyder SCarns PHarms KRoss RLockwood GWright N(2016)Modular HPC I/O characterization with DarshanProceedings of the 5th Workshop on Extreme-Scale Programming Tools10.5555/3018823.3018825(9-17)Online publication date: 13-Nov-2016
https://dl.acm.org/doi/10.5555/3018823.3018825
Malakar PVishwanath VKnight CMunson TPapka MWest J(2016)Optimal execution of co-analysis for large-scale molecular dynamics simulationsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.5555/3014904.3014985(1-14)Online publication date: 13-Nov-2016
https://dl.acm.org/doi/10.5555/3014904.3014985
Götz MBodenstein CRiedel M(2015)HPDBSCANProceedings of the Workshop on Machine Learning in High-Performance Computing Environments10.1145/2834892.2834894(1-10)Online publication date: 15-Nov-2015
https://dl.acm.org/doi/10.1145/2834892.2834894
Kettimuthu RVardoyan GAgrawal GSadayappan PFoster IKern JVetter J(2015)An elegant sufficiencyProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/2807591.2807660(1-12)Online publication date: 15-Nov-2015
https://dl.acm.org/doi/10.1145/2807591.2807660
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten