Abstract
Hyperparameter optimization is a crucial task in numerous applications of numerical modelling techniques. Methods as diverse as classical simulations and the great variety of machine learning techniques used nowadays, require an appropriate choice of their hyperparameters (HPs). While for classical simulations, calibration to measured data by numerical optimization techniques has a long tradition, the HPs of neural networks are often chosen by a mixture of grid search, random search and manual tuning.
In the present study the expert tool “OmniOpt” is introduced, which allows to optimize the HPs of a wide range of problems, ranging from classical simulations to different kinds of neural networks. Thereby, the emphasis is on versatility and flexibility for the user in terms of the applications and the choice of its HPs to be optimized. Moreover, the optimization procedure – which is usually a very time-consuming task – should be performed in a highly parallel way on the HPC system Taurus at TU Dresden. To this end, a Bayesian stochastic optimization algorithm (TPE) has been implemented on the Taurus system and connected to a user-friendly graphical user interface (GUI). In addition to the automatic optimization service, there is a variety of tools for analyzing and graphically displaying the results of the optimization.
The application of OmniOpt to a practical problem from material science is presented as an example.
This work was supported by the German Federal Ministry of Education and Research (BMBF, 01/S18026A-F) by funding the competence center for Big Data and AI “ScaDS.AI Dresden/Leipzig”.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning. TSSCML, pp. 3–33. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05318-5_1
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13, 281–305 (2012)
Shahriari, B., Swersky, K., Wang, Z., Adams, R., de Freitas, N.: Taking the human out of the loop: a review of Bayesian optimization. Proc. IEEE 104(1), 148–175 (2016)
Bergstra, J., et al.: Hyperopt: a Python library for model selection and hyperparameter optimization. Comput. Sci. Discov. 8, 014008 (2015). https://doi.org/10.1088/1749-4699/8/1/014008
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., Hutter, F.: Efficient and robust automated machine learning. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 28, pp. 2962–2970. Curran Associates, Inc. (2015)
Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J.E., Stocia, I.: Tune: a research platform for distributed model selection and training. arXiv:1807.05118 (2018)
Moritz, P., et al.: Ray: a distributed framework for emerging AI applications. arXiv:1712.05889 (2017)
Rapin, J., Teytaud O.: Nevergrad – a gradient-free optimization platform, GitHub repository (2018). https://GitHub.com/FacebookResearch/Nevergrad
Sergeev A., Del Balso, M.: Horovod: fast and easy distributed deep learning in TensorFlow. arXiv:1802.05799 (2018)
ZIH homepage. https://tu-dresden.de/zih/hochleistungsrechnen/hpc
Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Advances in Neural Information Processing Systems, vol. 24 (2011). https://papers.nips.cc/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf
Bullx documentation. https://www.dkrz.de/pdfs/docs/docu-mistral/bullx_scs_4_r4_de_2014-01.pdf
Yoo, A.B., Jette, M.A., Grondona, M.: SLURM: simple linux utility for resource management. https://doi.org/10.1007/10968987
MongoDB homepage. https://www.mongodb.com/
Zscheyge, M., Böhm, R., Hornig, A., Gerritzen, J., Gude, M.: Rate dependent non-linear mechanical behaviour of continuous fibre-reinforced thermoplastic composites – experimental characterisation and viscoelastic-plastic damage modelling. Mater. Des. 193, 108827 (2020)
Böhm, R., Gude, M., Hufenbach, W.: A phenomenologically based damage model for textile composites with crimped reinforcement. Comput. Sci. Technol. 70, 81–87 (2010)
Gude, M., Hufenbach, W., Ebert, C.: The strain-rate-dependent material and failure behaviour of 2D and 3D non-crimp glass-fibre-reinforced composites. Mech. Compos. Mater. 45, 467 (2009). https://doi.org/10.1007/s11029-009-9108-3
Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/
Heinrich, J., Weiskopf, D.: State of the art of parallel coordinates. In: Sbert, M., Szirmay-Kalos, L. (eds.) Eurographics 2013 - State of the Art Reports, pp. 95–116 (2013). https://doi.org/10.2312/conf/EG2013/stars/095-116
Acknowledgments
The authors would like to thank Taras Lazariv for his feedback and support which helped to improve this work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Winkler, P., Koch, N., Hornig, A., Gerritzen, J. (2021). OmniOpt – A Tool for Hyperparameter Optimization on HPC. In: Jagode, H., Anzt, H., Ltaief, H., Luszczek, P. (eds) High Performance Computing. ISC High Performance 2021. Lecture Notes in Computer Science(), vol 12761. Springer, Cham. https://doi.org/10.1007/978-3-030-90539-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-90539-2_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-90538-5
Online ISBN: 978-3-030-90539-2
eBook Packages: Computer ScienceComputer Science (R0)