Abstract
Linux container technology has more than proved itself useful in cloud computing as a lightweight alternative to virtualisation, whilst still offering good enough resource isolation. Docker is emerging as a popular runtime for managing Linux containers, providing both management tools and a simple file format. Research into the performance of containers compared to traditional Virtual Machines and bare metal shows that containers can achieve near native speeds in processing, memory and network throughput. A technology born in the cloud, it is making inroads into scientific computing both as a format for sharing experimental applications and as a paradigm for cloud based execution. However, it has unexplored uses in traditional cluster and grid computing. It provides a run time environment in which there is an opportunity for typical cluster and parallel applications to execute at native speeds, whilst being bundled with their own specific (or legacy) library versions and support software. This offers a solution to the Achilles heel of cluster and grid computing that requires the user to hold intimate knowledge of the local software infrastructure. Using Docker brings us a step closer to more effective job and resource management within the cluster by providing both a common definition format and a repeatable execution environment. In this paper we present the results of our work in deploying Docker containers in the cluster environment and an evaluation of its suitability as a runtime for high performance parallel execution. Our findings suggest that containers can be used to tailor the run time environment for an MPI application without compromising performance, and would provide better Quality of Service for users of scientific computing.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Blas (basic linear algebra subprograms). http://www.netlib.org/blas/
Docker. https://www.docker.com/
Dockerfile reference - docker documentation, Version 1.4. https://docs.docker.com/reference/builder/
Intel math kernel library linpack download. https://software.intel.com/en-us/articles/intel-math-kernel-library-linpack-download
Kernel based virtual machine. http://www.linux-kvm.org/
MPICH high performance portable MPI. http://www.mpich.org/
Open MPI: Open source high performance computing. http://www.open-mpi.org/
Charlie, C.: Standards for grid computing: global grid forum. J. Grid Comput. 1(1), 3–7 (2003)
Dooley, R.: Agave docker quickstart (2014). https://bitbucket.org/deardooley/agave-docker-support/
Dooley, R., Vaughn, M., Stanzione, D., Terry, S., Skidmore, E.: Software-as-a-service: the iplant foundation API. In: 5th IEEE Workshop on Many-Task Computing on Grids and Supercomputers, November 2012
Emeneker, W., Stanzione, D.: Dynamic virtual clustering. In: 2007 IEEE International Conference on Cluster Computing, pp. 84–90, September 2007
Felter, W., Ferreira, A., Rajamony, R., Rubio, J.: An updated performance comparison of virtual machines and linux containers. Technology 28, 32 (2014)
Kerrisk, M.: Namespaces in operation, part 1: namespaces overview (2014). http://lwn.net/Articles/531114/. Accessed 7 February 2015
Menage, P.: Cgroups. https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt. Accessed 7 February 2015
Smith, J., Nair, R.: Virtual Machines: Versatile Platforms for Systems and Processes. Morgan Kaufmann, San Francisco (2005)
Tannenbaum, T.: Htcondor and hep partnership and activities (2014). Presented at the HEPiX Fall 2014 Workshop. University of Nebraska, Lincoln, 13–17 October 2014
Vallee, G., Naughton, T., Scott, S.L.: System management software for virtual environments. In: Proceedings of the 4th International Conference on Computing Frontiers, pp. 153–160. ACM (2007)
Weiss, A.: Computing in the clouds. netWorker 11(4), 16–25 (2007)
Acknowledgments
The experimental results for this work could not have been obtained without the resources and support provided by the QueensGate Grid (QGG) at The University of Huddersfield.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Higgins, J., Holmes, V., Venters, C. (2015). Orchestrating Docker Containers in the HPC Environment. In: Kunkel, J., Ludwig, T. (eds) High Performance Computing. ISC High Performance 2015. Lecture Notes in Computer Science(), vol 9137. Springer, Cham. https://doi.org/10.1007/978-3-319-20119-1_36
Download citation
DOI: https://doi.org/10.1007/978-3-319-20119-1_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-20118-4
Online ISBN: 978-3-319-20119-1
eBook Packages: Computer ScienceComputer Science (R0)