research-article

Deploying Scientific Al Networks at Petaflop Scale on Secure Large Scale HPC Production Systems with Containers

Authors:

David Brayford,

Sofia VallecorsaAuthors Info & Claims

PASC '20: Proceedings of the Platform for Advanced Scientific Computing Conference

Article No.: 6, Pages 1 - 8

https://doi.org/10.1145/3394277.3401850

Published: 29 June 2020 Publication History

Abstract

There is an ever-increasing need for computational power to train complex artificial intelligence (AI) and machine learning (ML) models to tackle large scientific problems. High performance computing (HPC) resources are required to efficiently compute and scale complex models across tens of thousands of compute nodes. In this paper, we discuss the issues associated with the deployment of machine learning frameworks on large scale secure HPC systems and how we successfully deployed a standard machine learning framework on a secure large scale HPC production system, to train a complex three-dimensional convolutional GAN (3DGAN), with petaflop performance. 3DGAN is an example from the high energy physics domain, designed to simulate the energy pattern produced by showers of secondary particles inside a particle detector on various HPC systems.

References

[1]

Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 3 (2016).

[2]

D. Brayford. 2019. High Performance AI. Retrieved April 15, 2020 from https://github.com/DavidBrayford/HPAI

[3]

David Brayford, Sofia Vallecorsa, Atanas Atanasov, Fabio Baruffa, and Walter Riviera. 2019. Deploying AI Frameworks on Secure HPC Systems with Containers. In 2019 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 1--6.

[4]

F. Carmianti, G. K., S. Vallecorsa, M. Cai, V. Codreanu, D. Podareanu, A. Atanasov, F. Baruffa, S. Choi, V. Saletore, H. Pabst, W. Riviera, and D. Brayford. 2020. Generative Adversarial Networks for Fast Simulation. Retrieved April 15, 2020 from https://indico.cern.ch/event/853334/contributions/3706456/attachments/1973668/3284005/3DGAN.pdf

[5]

Federico Carminati, Andrei Gheata, Gulrukh Khattak, P Mendez Lorenzo, S Sharan, and S Vallecorsa. 2018. Three dimensional Generative Adversarial Networks for fast simulation. In Journal of Physics: Conference Series, Vol. 1085. IOP Publishing, 032016.

[6]

Federico Carminati, Gulrukh Khattak, Maurizio Pierini, Amir Farbin, Benjamin Hooberman, Wei Wei, Matt Zhang, Vitória Barin Pacela, Sofia Vallecorsa, Maria Spiropulu, and Jean-Roch Vlimant. 2017. Calorimetry with deep learning: particle classification, energy regression, and simulation for high-energy physics. In NIPS.

[7]

F. Chollet. 2015. Keras.

[8]

Valeriu Codreanu, Damian Podareanu, and Vikram Saletore. 2017. Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train. arXiv preprint arXiv:1711.04291 (2017).

[9]

CLIC collaboration et al. 2012. CLIC conceptual design report. Technical Report. CERN-2012-007.

[10]

Leonardo Dagum and Ramesh Menon. 1998. OpenMP: an industry standard API for shared-memory programming. IEEE computational science and engineering 5, 1 (1998), 46--55.

[11]

Jack J Dongarra. 1987. TheLINPACKbenchmark:Anexplanation. In International Conference on Supercomputing. Springer, 456--474.

[12]

Evangelos Georganas, Sasikanth Avancha, Kunal Banerjee, Dhiraj Kalamkar, Greg Henry, Hans Pabst, and Alexander Heinecke. 2018. Anatomy of high-performance deep learning convolutions on simd architectures. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 830--841.

Digital Library

[13]

Lisa Gerhardt, Wahid Bhimji, Markus Fasel, Jeff Porter, Mustafa Mustafa, Doug Jacobsen, Vakho Tsulaia, and Shane Canon. 2017. Shifter: Containers for HPC. In J. Phys. Conf. Ser., Vol. 898. 082021.

[14]

Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Y Bengio. 2014. Generative adversarial nets, In Proceedings of the 27th International Conference on Neural Information Processing Systems. Ghahramani Z, editor, 2672--2680.

[15]

Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. 2012. Lecture 6a overview of mini-batch gradient descent. Coursera Lecture slides https://cla.ss.coursera.org/neuralnets-2012-001/lecture,[Online (2012).

[16]

Intel. 2020. Intel Python Distribution Benchmarks.

[17]

intelaipg. [n.d.]. IntelAI. Retrieved April 15, 2020 from https://hub.docker.com/u/intelaipg

[18]

Thorsten Kurth, Sean Treichler, Joshua Romero, Mayur Mudigonda, Nathan Luehr, Everett Phillips, Ankur Mahesh, Michael Matheson, Jack Deslippe, Massimiliano Fatica, et al. 2018. Exascale deep learning for climate analytics. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 649--660.

Digital Library

[19]

Gregory M Kurtzer, Vanessa Sochat, and Michael W Bauer. 2017. Singularity: Scientific containers for mobility of compute. PloS one 12, 5 (2017).

[20]

Yunjie Liu, Evan Racah, Joaquin Correa, Amir Khosrowshahi, David Lavers, Kenneth Kunkel, Michael Wehner, William Collins, et al. 2016. Application of deep convolutional neural networks for detecting extreme weather in climate datasets. arXiv preprint arXiv:1605.01156 (2016).

[21]

Dirk Merkel. 2014. Docker: lightweight linux containers for consistent development and deployment. Linux Journal 2014, 239 (2014), 2.

Digital Library

[22]

Mayur Mudigonda, Sookyung Kim, Ankur Mahesh, Samira Kahou, Karthik Kashinath, Dean Williams, Vincent Michalski, Travis O'Brien, and Mr Prabhat. 2017. Segmenting and tracking extreme climate events using neural networks. In Deep Learning for Physical Sciences (DLPS) Workshop, held with NIPS Conference.

[23]

Augustus Odena, Christopher Olah, and Jonathon Shlens. 2017. Conditional image synthesis with auxiliary classifier gans. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2642--2651.

Digital Library

[24]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).

[25]

podman.io. [n.d.]. Podman. Retrieved April 15, 2020 from https://podman.io

[26]

Reid Priedhorsky and Tim Randles. 2017. Charliecloud: Unprivileged containers for user-defined software stacks in hpc. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1--10.

Digital Library

[27]

Evan Racah, Christopher Beckham, Tegan Maharaj, Samira Ebrahimi Kahou, Mr Prabhat, and Chris Pal. 2017. ExtremeWeather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events. In Advances in Neural Information Processing Systems. 3402--3413.

Digital Library

[28]

G. Rossum. 1995. Python reference manual. CWI (Centre for Mathematics and Computer Science).

[29]

Karl W Schulz, C Reese Baird, David Brayford, Yiannis Georgiou, Gregory M Kurtzer, Derek Simmel, Thomas Sterling, Nirmala Sundararajan, and Eric Van Hensbergen. 2016. Cluster computing with OpenHPC. (2016).

[30]

Alexander Sergeev and Mike Del Balso. 2018. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799 (2018).

[31]

Alfred Torrez, Timothy Randles, and Reid Priedhorsky. 2019. HPC container runtimes have minimal or no performance impact. In 2019 IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC). IEEE, 37--42.

[32]

Sofia Vallecorsa, Federico Carminati, Gulrukh Khattak, Damian Podareanu, Valeriu Codreanu, Vikram Saletore, and Hans Pabst. 2018. Distributed training of generative adversarial networks for fast detector simulation. In International Conference on High Performance Computing. Springer, 487--503.

Cited By

Zhu SYu TXu TChen HDustdar SGigan SGunduz DHossain EJin YLin FLiu BWan ZZhang JZhao ZZhu WChen ZDurrani TWang HWu JZhang TPan Y(2023)Intelligent Computing: The Latest Advances, Challenges, and FutureIntelligent Computing10.34133/icomputing.00062Online publication date: 30-Jan-2023
https://doi.org/10.34133/icomputing.0006
Diaz-de-Arcaya JTorre-Bastida AZárate GMiñón RAlmeida A(2023)A Joint Study of the Challenges, Opportunities, and Roadmap of MLOps and AIOps: A Systematic SurveyACM Computing Surveys10.1145/362528956:4(1-30)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3625289
Yao TWang JWan MXin ZWang YCao RLi SChi X(2022)VenusAIJournal of Systems Architecture: the EUROMICRO Journal10.1016/j.sysarc.2022.102550128:COnline publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1016/j.sysarc.2022.102550
Show More Cited By

Index Terms

Deploying Scientific Al Networks at Petaflop Scale on Secure Large Scale HPC Production Systems with Containers
1. Computing methodologies
  1. Artificial intelligence
    1. Distributed artificial intelligence
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Software system models
        Massively parallel systems

Recommendations

Dynamic steering of HPC scientific workflows

The field of scientific workflow management systems has grown significantly as applications start using them successfully. In 2007, several active researchers in scientific workflow developments presented the challenges for the state of the art in ...
Computing on large-scale distributed systems: Xtrem Web architecture, programming models, security, tests and convergence with grid
Special issue: P2P computing and interaction with grids

Global Computing systems belong to the class of large-scale distributed systems. Their properties high computational, storage and communication performance potentials, high resilience make them attractive in academia and industry as computing ...
Ensemble Prediction of Job Resources to Improve System Performance for Slurm-Based HPC Systems
PEARC '21: Practice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions

In this paper, we present a novel methodology for predicting job resources (memory and time) for submitted jobs on HPC systems. Our methodology based on historical jobs data (saccount data) provided from the Slurm workload manager using supervised ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

PASC '20: Proceedings of the Platform for Advanced Scientific Computing Conference

June 2020

169 pages

ISBN:9781450379939

DOI:10.1145/3394277

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing
CSCS: Swiss National Supercomputing Centre
ETH Zurich: Federal Institute of Technology - University of Zurich

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

PASC '20

Sponsor:

SIGHPC
CSCS
ETH Zurich

PASC '20: Platform for Advanced Scientific Computing Conference

June 29 - July 1, 2020

Geneva, Switzerland

Acceptance Rates

PASC '20 Paper Acceptance Rate 16 of 36 submissions, 44%;

Overall Acceptance Rate 109 of 221 submissions, 49%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
163
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)2

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhu SYu TXu TChen HDustdar SGigan SGunduz DHossain EJin YLin FLiu BWan ZZhang JZhao ZZhu WChen ZDurrani TWang HWu JZhang TPan Y(2023)Intelligent Computing: The Latest Advances, Challenges, and FutureIntelligent Computing10.34133/icomputing.00062Online publication date: 30-Jan-2023
https://doi.org/10.34133/icomputing.0006
Diaz-de-Arcaya JTorre-Bastida AZárate GMiñón RAlmeida A(2023)A Joint Study of the Challenges, Opportunities, and Roadmap of MLOps and AIOps: A Systematic SurveyACM Computing Surveys10.1145/362528956:4(1-30)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3625289
Yao TWang JWan MXin ZWang YCao RLi SChi X(2022)VenusAIJournal of Systems Architecture: the EUROMICRO Journal10.1016/j.sysarc.2022.102550128:COnline publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1016/j.sysarc.2022.102550
González-Abad JLópez García ÁKozlov V(2022)A container-based workflow for distributed training of deep learning algorithms in HPC clustersCluster Computing10.1007/s10586-022-03798-726:5(2815-2834)Online publication date: 7-Nov-2022
https://doi.org/10.1007/s10586-022-03798-7
Brayford DAllalen MIapichino LBrennan JMoran NQ'Riordan LHanley K(2021)Deploying Containerized QuanEX Quantum Simulation Software on HPC Systems2021 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)10.1109/CANOPIEHPC54579.2021.00005(1-9)Online publication date: Nov-2021
https://doi.org/10.1109/CANOPIEHPC54579.2021.00005
Ruhela AHarrell SEvans RZynda GFonner JVaughn MMinyard TCazes J(2021)Characterizing Containerized HPC Applications Performance at Petascale on CPU and GPU ArchitecturesHigh Performance Computing10.1007/978-3-030-78713-4_22(411-430)Online publication date: 24-Jun-2021
https://dl.acm.org/doi/10.1007/978-3-030-78713-4_22

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents