research-article

Nix as HPC package management system

Authors:

Oliver Henriot,

Olivier Richard,

Laure TavardAuthors Info & Claims

HUST'17: Proceedings of the Fourth International Workshop on HPC User Support Tools

Article No.: 4, Pages 1 - 6

https://doi.org/10.1145/3152493.3152556

Published: 12 November 2017 Publication History

Abstract

Modern High Performance Computing systems are becoming larger and more heterogeneous. The proper management of software for the users of such systems poses a significant challenge. These users run very diverse applications that may be compiled with proprietary tools for specialized hardware. Moreover, the application life-cycle of these software may exceed the lifetime of the HPC systems themselves. These difficulties motivate the use of specialized package management systems. In this paper, we outline an approach to HPC package development, deployment, management, sharing, and reuse based on the Nix functional package manager. We report our experience with this approach inside the GRICAD HPC center[GRICAD 2017a] in Grenoble over a 12 month period and compare it to other existing approaches.

References

[1]

Dong H. Ahn. 2014. Flux: A Next-Generation Resource Management Framework for Large HPC Centers. (2014). Retrieved July 28, 2017 from https://flux-framework.github.io/papers/Flux-SRMPDS-final.pdf

[2]

Susan Baldwin. 2012. Compute Canada: advancing computational research. In Journal of Physics: Conference Series, Vol. 341. IOP Publishing, 012001.

[3]

Daniel Balouek, Alexandra Carpen Amarie, Ghislain Charrier, Frédéric Desprez, Emmanuel Jeannot, Emmanuel Jeanvoine, Adrien Lèbre, David Margery, Nicolas Niclausse, Lucas Nussbaum, Olivier Richard, Christian Pérez, Flavien Quesnel, Cyril Rohr, and Luc Sarzyniec. 2013. Adding Virtualization Capabilities to the Grid'5000 Testbed. In Cloud Computing and Services Science, Ivan I. Ivanov, Marten van Sinderen, Frank Leymann, and Tony Shan (Eds.). Communications in Computer and Information Science, Vol. 367. Springer International Publishing, 3--20. https://doi.org/10.1007/978-3-319-04519-1_1

[4]

Lucas Bruno. 2017. Lethalman Blog. (2017). Retrieved July 28, 2017 from http://lethalman.blogspot.fr/2014/07/nix-pill-1-why-you-should-give-it-try.html

[5]

Bruno Bzeznik. 2017. Setting up NIX on a multi-users HPC environment. (May 2017). Retrieved July 28, 2017 from https://gricad.github.io/calcul/nix/hpc/2017/05/15/nix-on-hpc-platforms.html

[6]

Nixpkgs contributors. Retrieved 2017. NetCDF software component. (Retrieved 2017). Retrieved July 28, 2017 from https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/netcdf-cxx4/default.nix

[7]

Ludovic Courtès and Ricardo Wurmus. 2015. Reproducible and User-Controlled Software Environments in HPC with Guix. In 2nd International Workshop on Reproducibility in Parallel Computing (RepPar). Vienne, Austria. https://hal.inria.fr/hal-01161771

[8]

Eelco Dolstra, Andres Lãűh, and Nicolas Pierron. 2010. NixOS: A Purely Functional Linux Distribution. Journal of Functional Programming. 20, 5--6 (2010), 577--615. https://doi.org/10.1017/S0956796810000195

Digital Library

[9]

Eelco Dosltra. 2006. The Purely Functional Software Deployment Model. Ph.D. Dissertation. University. Retrieved July 28, 2017 from http://nixos.org/~eelco/pubs/phd-thesis.pdf

[10]

EasyBuild. 2017. EasyBuild: building software with ease. (2017). Retrieved July 28, 2017 from https://easybuilders.github.io/easybuild/

[11]

Melara et al. 2017. Using Spack to Manage Software on Cray Supercomputers. (2017). Retrieved July 28, 2017 from https://cug.org/proceedings/cug2017_proceedings/includes/files/pap153s2-file1.pdf

[12]

GRICAD. 2017a. GRAICAD Web site. (2017). Retrieved July 28, 2017 from hgj.

[13]

GRICAD. 2017b. GRICAD Ciment packages collection. (May 2017). Retrieved July 28, 2017 from https://github.com/Gricad/nix-ciment-channel

[14]

Guix. 2017. The Guix System Distribution. (2017). Retrieved July 28, 2017 from https://www.gnu.org/software/guix/

[15]

LIG/INRIA. 2003. OAR Website. (2003). Retrieved July 28, 2017 from https://oar.imag.fr/

[16]

Hans Werner Meuer, Erich Strohmaier, Jack Dongarra, and Horst D. Simon. 2014. The TOP500: History, Trends, and Future Directions in High Performance Computing (1st ed.). Chapman & Hall/CRC.

[17]

Nix. 2017a. Nix packages collection. (2017). Retrieved July 28, 2017 from https://github.com/NixOS/nixpkgs

[18]

Nix. 2017b. Nix. The Purely Functional Package Manager. (2017). Retrieved July 28, 2017 from https://nixos.org/nix/

[19]

SchedMD. 2003. Slurm Website. (2003). Retrieved July 28, 2017 from https://slurm.schedmd.com/

[20]

Spack. 2017. A flexible package manager that supports multiple versions, configurations, platforms, and compilers. (2017). Retrieved July 28, 2017 from https://spack.io/

Cited By

Grayson SMarinov DKatz DMilewicz R(2023)Automatic Reproduction of Workflows in the Snakemake Workflow Catalog and nf-core RegistriesProceedings of the 2023 ACM Conference on Reproducibility and Replicability10.1145/3589806.3600037(74-84)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3589806.3600037
Gollenstede NKulau UDietrich CEgger BLee D(2023)reUpNix: Reconfigurable and Updateable Embedded SystemsProceedings of the 24th ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems10.1145/3589610.3596273(40-51)Online publication date: 13-Jun-2023
https://dl.acm.org/doi/10.1145/3589610.3596273
Liu FRahaman RWeiner MCoulter EPhanish DValdez JSarajlic SLara RBuffington P(2023)Semi-Automatic Hybrid Software Deployment Workflow in a Research Computing CenterPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good10.1145/3569951.3593607(68-74)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3569951.3593607
Show More Cited By

Nix as HPC package management system
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains

Recommendations

NixOS: a purely functional Linux distribution
ICFP '08: Proceedings of the 13th ACM SIGPLAN international conference on Functional programming

Existing package and system configuration management tools suffer from an imperative model, where system administration actions such as upgrading packages or changes to system configuration files are stateful: they destructively update the state of the ...
Package Management Systems

A package management system organizes and simplifies the installation and maintenance of software by standardizing and organizing the production and consumption of software collections. As a software developer, you can benefit from package managers in ...
The HPC Testbed of the Italian Grid Infrastructure
PDP '13: Proceedings of the 2013 21st Euromicro International Conference on Parallel, Distributed, and Network-Based Processing

Even though the Italian Grid Infrastructure (IGI) is a general purpose distributed platform, in the past it has been used mainly for serial computations. Parallel applications have been typically executed on supercomputer facilities or, in case of ``not ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

HUST'17: Proceedings of the Fourth International Workshop on HPC User Support Tools

November 2017

44 pages

ISBN:9781450351300

DOI:10.1145/3152493

Copyright © 2017 ACM.

© 2017 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Labex

Conference

SC '17

Sponsor:

SIGHPC

SC '17: The International Conference for High Performance Computing, Networking, Storage and Analysis

November 12 - 17, 2017

CO, Denver, USA

Acceptance Rates

Overall Acceptance Rate 5 of 9 submissions, 56%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
412
Total Downloads

Downloads (Last 12 months)44
Downloads (Last 6 weeks)3

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Grayson SMarinov DKatz DMilewicz R(2023)Automatic Reproduction of Workflows in the Snakemake Workflow Catalog and nf-core RegistriesProceedings of the 2023 ACM Conference on Reproducibility and Replicability10.1145/3589806.3600037(74-84)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3589806.3600037
Gollenstede NKulau UDietrich CEgger BLee D(2023)reUpNix: Reconfigurable and Updateable Embedded SystemsProceedings of the 24th ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded Systems10.1145/3589610.3596273(40-51)Online publication date: 13-Jun-2023
https://dl.acm.org/doi/10.1145/3589610.3596273
Liu FRahaman RWeiner MCoulter EPhanish DValdez JSarajlic SLara RBuffington P(2023)Semi-Automatic Hybrid Software Deployment Workflow in a Research Computing CenterPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good10.1145/3569951.3593607(68-74)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3569951.3593607
Sly-Delgado BLocascio NSimonetti DWiseman BTovar BThain DBabuji YChard KFoster ILi Z(2022)PONCHOProceedings of the 2nd Workshop on High Performance Serverless Computing10.1145/3526060.3535459(8-14)Online publication date: 30-Jun-2022
https://dl.acm.org/doi/10.1145/3526060.3535459
Jeter TBobbitt MRountree B(2022)SpackNVD: A Vulnerability Audit Tool for Spack Packages2022 IEEE/ACM First International Workshop on Cyber Security in High Performance Computing (S-HPC)10.1109/S-HPC56715.2022.00007(9-17)Online publication date: Nov-2022
https://doi.org/10.1109/S-HPC56715.2022.00007
Goswami RS. RGoswami AGoswami SGoswami D(2022)Reproducible High Performance Computing without Redundancy with Nix2022 Seventh International Conference on Parallel, Distributed and Grid Computing (PDGC)10.1109/PDGC56933.2022.10053342(238-242)Online publication date: 25-Nov-2022
https://doi.org/10.1109/PDGC56933.2022.10053342
Dahm JRichards DBlack ABertsch AGrinberg LKarlin IKokkila-Schumacher SLeon ENeely JPankajakshan RPearce O(2020)Sierra Center of Excellence: Lessons learnedIBM Journal of Research and Development10.1147/JRD.2019.296106964:3/4(2:1-2:14)Online publication date: 1-May-2020
https://doi.org/10.1147/JRD.2019.2961069
Vaillancourt PWineholt BBarker BDeliyannis PZheng JSuresh ABrazier AKnepper RWolski R(2020)Reproducible and Portable Workflows for Scientific Computing and HPC in the CloudPractice and Experience in Advanced Research Computing 2020: Catch the Wave10.1145/3311790.3396659(311-320)Online publication date: 26-Jul-2020
https://dl.acm.org/doi/10.1145/3311790.3396659
Mehringer SBarker B(2020)Using Containers to Create More Interactive Online Training and Education MaterialsPractice and Experience in Advanced Research Computing 2020: Catch the Wave10.1145/3311790.3396641(246-251)Online publication date: 26-Jul-2020
https://dl.acm.org/doi/10.1145/3311790.3396641
Vaillancourt PCoulter JKnepper RBarker B(2020)Self-Scaling Clusters and Reproducible Containers to Enable Scientific Computing2020 IEEE High Performance Extreme Computing Conference (HPEC)10.1109/HPEC43674.2020.9286208(1-8)Online publication date: 22-Sep-2020
https://doi.org/10.1109/HPEC43674.2020.9286208
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents