Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

HyperTuner: a cross-layer multi-objective hyperparameter auto-tuning framework for data analytic services

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Hyperparameters optimization (HPO) is vital for machine learning models. Besides model accuracy, other tuning intentions such as model training time and energy consumption are also worthy of attention from data analytic service providers. Therefore, it is essential to take both model hyperparameters and system parameters into consideration to execute cross-layer multi-objective hyperparameter auto-tuning. Toward this challenging target, we propose HyperTuner in this paper which leverages a well-designed ADUMBO algorithm to find the Pareto-optimal configuration set. Compared with vanilla Bayesian optimization-based methods, ADUMBO selects the most promising configuration from the generated Pareto candidate set during each iteration via maximizing a novel adaptive uncertainty metric. We evaluate HyperTuner on our local distributed TensorFlow cluster, and experimental results show that it is always able to find a better Pareto configuration front superior in both convergence and diversity compared with the other four baseline algorithms. Besides, experiments with different training datasets, different optimization objectives, and different machine learning platforms verify that HyperTuner can well adapt to various data analytic service scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data and materials availability

All of the material is owned by the authors and/or no permissions are required. Data will be made available on request.

References

  1. Pouyanfar S, Sadiq S, Yan Y, Tian H, Tao Y, Reyes MP, Shyu M-L, Chen S-C, Iyengar SS (2018) A survey on deep learning: algorithms, techniques, and applications. ACM Comput Surv (CSUR) 51(5):1–36

    Article  Google Scholar 

  2. Pang G, Shen C, Cao L, Hengel AVD (2021) Deep learning for anomaly detection: a review. ACM Comput Surv (CSUR) 54(2):1–38

    Article  Google Scholar 

  3. Kotthoff L, Thornton C, Hoos HH, Hutter F, Leyton-Brown K (2019) Auto-weka: Automatic model selection and hyperparameter optimization in weka. Autom Mach Learn Meth Syst Challenges, 81–95

  4. Li L, Jamieson K, DeSalvo G, Rostamizadeh A, Talwalkar A (2017) Hyperband: a novel bandit-based approach to hyperparameter optimization. J Mach Learn Res 18(1):6765–6816

    MathSciNet  Google Scholar 

  5. Falkner S, Klein A, Hutter F (2018) Bohb: Robust and efficient hyperparameter optimization at scale. In: International Conference on Machine Learning, pp. 1437–1446. PMLR

  6. Akiba T, Sano S, Yanase T, Ohta T, Koyama M (2019) Optuna: A next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2623–2631

  7. Strubell E, Ganesh A, McCallum A (2019) Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243

  8. Morales-Hernández A, Van Nieuwenhuyse I, Rojas Gonzalez S (2022) A survey on multi-objective hyperparameter optimization algorithms for machine learning. Artif Intell Rev 1–51

  9. Smithson SC, Yang G, Gross WJ, Meyer BH (2016) Neural networks designing neural networks: multi-objective hyper-parameter optimization. In: 2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1–8. IEEE

  10. Zuluaga M, Sergent G, Krause A, Püschel M (2013) Active learning for multi-objective optimization. In: International Conference on Machine Learning, pp. 462–470. PMLR

  11. Hernández-Lobato D, Hernandez-Lobato J, Shah A, Adams R (2016) Predictive entropy search for multi-objective bayesian optimization. In: International Conference on Machine Learning, pp. 1492–1501. PMLR

  12. Emmerich MT, Giannakoglou KC, Naujoks B (2006) Single-and multiobjective evolutionary optimization assisted by gaussian random field metamodels. IEEE Trans Evolut Comput 10(4):421–439

    Article  Google Scholar 

  13. Ponweiser W, Wagner T, Biermann D, Vincze M (2008) Multiobjective optimization on a limited budget of evaluations using model-assisted \(\{\)S\(\}\) -metric selection. In: International Conference on Parallel Problem Solving from Nature, pp. 784–794. Springer

  14. Iqbal MS, Su J, Kotthoff L, Jamshidi P (2020) Flexibo: Cost-aware multi-objective optimization of deep neural networks. arXiv preprint arXiv:2001.06588

  15. Laumanns M, Thiele L, Deb K, Zitzler E (2002) Combining convergence and diversity in evolutionary multiobjective optimization. Evolut Comput 10(3):263–282

    Article  Google Scholar 

  16. Jiang S, Yang S (2016) Convergence versus diversity in multiobjective optimization. In: Parallel Problem Solving from Nature–PPSN XIV: 14th International Conference, Edinburgh, UK, September 17-21, 2016, Proceedings 14, pp. 984–993. Springer

  17. Hasabnis N (2018) Auto-tuning tensorflow threading model for cpu backend. In: 2018 IEEE/ACM Machine Learning in HPC Environments (MLHPC), pp. 14–25. IEEE

  18. Spantidi O, Galanis I, Anagnostopoulos I (2020) Frequency-based power efficiency improvement of cnns on heterogeneous iot computing systems. In: 2020 IEEE 6th World Forum on Internet of Things (WF-IoT), pp. 1–6. IEEE

  19. Tang Z, Wang Y, Wang Q, Chu X (2019) The impact of gpu dvfs on the energy and performance of deep learning: An empirical study. In: Proceedings of the Tenth ACM International Conference on Future Energy Systems, pp. 315–325

  20. Stamoulis D, Cai E, Juan D-C, Marculescu D (2018) Hyperpower: Power-and memory-constrained hyper-parameter optimization for neural networks. In: 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 19–24. IEEE

  21. Capra M, Bussolino B, Marchisio A, Masera G, Martina M, Shafique M (2020) Hardware and software optimizations for accelerating deep neural networks: survey of current trends, challenges, and the road ahead. IEEE Access 8:225134–225180

    Article  Google Scholar 

  22. Linux Kernel (2023) https://www.kernel.org/doc/html/v4.14/admin-guide/pm/cpufreq.html. Accessed on Feb 20,

  23. Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans Evolut Comput 6(2):182–197

    Article  Google Scholar 

  24. Srinivas N, Deb K (1994) Muiltiobjective optimization using nondominated sorting in genetic algorithms. Evolut Comput 2(3):221–248

    Article  Google Scholar 

  25. Zhang Q, Li H (2007) Moea/d: a multiobjective evolutionary algorithm based on decomposition. IEEE Trans Evolut Comput 11(6):712–731

    Article  Google Scholar 

  26. Magda M, Martinez-Alvarez A, Cuenca-Asensi S (2017) Mooga parameter optimization for onset detection in emg signals. In: New Trends in Image Analysis and Processing–ICIAP 2017: ICIAP International Workshops, WBICV, SSPandBE, 3AS, RGBD, NIVAR, IWBAAS, and MADiMa 2017, Catania, Italy, September 11-15, 2017, Revised Selected Papers 19, pp. 171–180. Springer

  27. Calisto MB, Lai-Yuen SK (2020) Adaen-net: an ensemble of adaptive 2d–3d fully convolutional networks for medical image segmentation. Neural Netw 126:76–94

    Article  Google Scholar 

  28. Bubeck S, Cesa-Bianchi N et al (2012) Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundat Trends Mach Learn 5(1):1–122

    Article  Google Scholar 

  29. Browne CB, Powley E, Whitehouse D, Lucas SM, Cowling PI, Rohlfshagen P, Tavener S, Perez D, Samothrakis S, Colton S (2012) A survey of monte carlo tree search methods. IEEE Trans Comput Intell AI Games 4(1):1–43

    Article  Google Scholar 

  30. Parsa M, Ankit A, Ziabari A, Roy K (2019) Pabo: Pseudo agent-based multi-objective bayesian hyperparameter optimization for efficient neural accelerator design. In: 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1–8. IEEE

  31. Belakaria S, Deshwal A, Jayakodi NK, Doppa JR (2020) Uncertainty-aware search framework for multi-objective Bayesian optimization. Proc AAAI Conf Artif Intell 34:10044–10052

    Google Scholar 

  32. Srinivas N, Krause A, Kakade SM, Seeger M (2009) Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995

  33. Ansible Playbook. https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html

  34. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  35. Cats vs (2023) Dogs. https://www.kaggle.com/c/dogs-vs-cats. Accessed on Feb 20,

  36. IMDB (2023) https://keras.io/api/datasets/imdb/. Accessed on Feb 20,

  37. Turbostat (2023) https://www.mankier.com/8/turbostat. Accessed on Feb 20

  38. Weymark JA (1981) Generalized gini inequality indices. Math Soc Sci 1(4):409–430

    Article  MathSciNet  Google Scholar 

  39. Shahriari B, Swersky K, Wang Z, Adams RP, De Freitas N (2015) Taking the human out of the loop: a review of Bayesian optimization. Proc IEEE 104(1):148–175

    Article  Google Scholar 

  40. Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13(2)

  41. Karl F, Pielok T, Moosbauer J, Pfisterer F, Coors S, Binder M, Schneider L, Thomas J, Richter J, Lang M, et al (2022) Multi-objective hyperparameter optimization–an overview. arXiv preprint arXiv:2206.07438

  42. Riquelme N, Von Lücken C, Baran B (2015) Performance metrics in multi-objective optimization. In: 2015 Latin American Computing Conference (CLEI), pp. 1–11. IEEE

  43. McKnight PE, Najab J (2010) Mann-whitney u test. The Corsini encyclopedia of psychology, 1–1

  44. Cohen G, Afshar S, Tapson J, Van Schaik A (2017) Emnist: Extending mnist to handwritten letters. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2921–2926. IEEE

  45. Montgomery DC (2017) Design and analysis of experiments. Wiley

  46. Hartikainen M, Miettinen K, Wiecek MM (2012) Paint: Pareto front interpolation for nonlinear multiobjective optimization. Comput Optimiz Appl 52(3):845–867

    Article  MathSciNet  Google Scholar 

  47. Knowles J (2006) Parego: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Trans Evolut Comput 10(1):50–66

    Article  Google Scholar 

  48. Zela A, Klein A, Falkne S, Hutter F (2018) Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint arXiv:1807.06906

  49. Capra M, Bussolino B, Marchisio A, Masera G, Martina M, Shafique M (2020) Hardware and software optimizations for accelerating deep neural networks: survey of current trends, challenges, and the road ahead. IEEE Access 8:225134–225180

    Article  Google Scholar 

  50. Nabavinejad SM, Reda S (2021) Bayestuner: Leveraging Bayesian optimization for DNN inference configuration selection. IEEE Comput Arch Lett 20(2):166–170

    Article  Google Scholar 

  51. Lokhmotov A, Chunosov N, Vella F, Fursin G (2018) Multi-objective autotuning of mobilenets across the full software/hardware stack. In: Proceedings of the 1st on Reproducible Quality-Efficient Systems Tournament on Co-designing Pareto-efficient Deep Learning, p. 1

Download references

Funding

This work was supported by the National Natural Science Foundation of China under Grant 61902440 and 62272001.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Anhui University, Hefei, 230601, China

    Hui Dou, Shanshan Zhu & Yiwen Zhang

  2. School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, 510006, China

    Pengfei Chen & Zibin Zheng

Authors
  1. Hui Dou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shanshan Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yiwen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pengfei Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zibin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Hui Dou helped in conceptualization, writing—review & editing. Shanshan Zhu was involved in software, validation, writing—original draft preparation. Yiwen Zhang (corresponding author) contributed to supervision, resources. Pengfei Chen contributed to methodology, formal analysis. Zibin Zheng administrated the project. All authors reviewed the manuscript.

Corresponding author

Correspondence to Yiwen Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest as defined by Springer, or other interests that might be perceived to influence the results and/or discussion reported in this paper.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dou, H., Zhu, S., Zhang, Y. et al. HyperTuner: a cross-layer multi-objective hyperparameter auto-tuning framework for data analytic services. J Supercomput 80, 17460–17491 (2024). https://doi.org/10.1007/s11227-024-06123-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-024-06123-8

Keywords

Search

Navigation