Abstract
Developing an efficient solver for hyperparameter optimization (HPO) can help to support the environmental sustainability of modern AI. One popular solver for HPO problems is called BOHB, which attempts to combine the benefits of Bayesian optimization (BO) and Hyperband. It conducts the sampling of configurations with the aid of a BO surrogate model. However, only the few high-fidelity measurements are utilized in the building of BO surrogate model, leading to the fact that the built BO surrogate cannot well model the objective function in HPO. Especially, in the scenario of multiobjective optimization (which is more complicated than single-objective optimization), the resultant BO surrogates for modelling all conflicting objective functions would be more likely to mislead the configuration search. To tackle this low-efficiency issue, in this paper, we propose an efficient algorithm, referred as Multiobjective Multi-Fidelity Bayesian Optimization and Hyperband, for solving multiobjective HPO problems. The key idea is to fully consider the contributions of computationally cheap low-fidelity surrogates and expensive high-fidelity surrogates, and enable effective utilization of the integrated information of multi-fidelity ensemble model in an online manner. The weightages for distinct fidelities are adaptively determined based on the approximation performance of their corresponding surrogates. A range of experiments on diversified real-world multiobjective HPO problems (including the HPO of multi-label/multi-task learning models and the HPO of models with several performance metrics) are carried out to investigate the performance of our proposed algorithm. Experimental results showcase that the proposed algorithm outperforms more than 10 state-of-the-art peers, while demonstrating the ability of our proposed algorithm to efficiently solve real-world multiobjective HPO problems at scale.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Anthony, L.F.W., Kanding, B., Selvan, R.: Carbontracker: tracking and predicting the carbon footprint of training deep learning models. arXiv preprint arXiv:2007.03051 (2020)
Belakaria, S., Deshwal, A.: Max-value entropy search for multi-objective Bayesian optimization. In: International Conference on Neural Information Processing Systems (NeurIPS) (2019)
Belakaria, S., Deshwal, A., Doppa, J.R.: Multi-fidelity multi-objective Bayesian optimization: an output space entropy search approach. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 10035–10043 (2020)
Belakaria, S., Deshwal, A., Jayakodi, N.K., Doppa, J.R.: Uncertainty-aware search framework for multi-objective bayesian optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 10044–10052 (2020)
Cai, H., Gan, C., Wang, T., Zhang, Z., Han, S.: Once-for-all: train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791 (2019)
Emmerich, M., Klinkenberg, J.W.: The computation of the expected improvement in dominated hypervolume of pareto front approximations. Technical report, Leiden University, p. 34 (2008)
Falkner, S., Klein, A., Hutter, F.: BOHB: robust and efficient hyperparameter optimization at scale. In: International Conference on Machine Learning, pp. 1437–1446. PMLR (2018)
Goel, T., Haftka, R.T., Shyy, W., Queipo, N.V.: Ensemble of surrogates. Struct. Multidiscip. Optim. 33(3), 199–216 (2007)
He, X., Zhao, K., Chu, X.: AutoML: a survey of the state-of-the-art. Knowl.-Based Syst. 212, 106622 (2021)
Hernández-Lobato, D., Hernandez-Lobato, J., Shah, A., Adams, R.: Predictive entropy search for multi-objective Bayesian optimization. In: International Conference on Machine Learning, pp. 1492–1501. PMLR (2016)
Hu, Y.Q., Yu, Y., Tu, W.W., Yang, Q., Chen, Y., Dai, W.: Multi-fidelity automatic hyper-parameter tuning via transfer series expansion. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 3846–3853 (2019)
Izquierdo, S., et al.: Bag of baselines for multi-objective joint neural architecture search and hyperparameter optimization. In: 8th ICML Workshop on Automated Machine Learning (AutoML) (2021)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Klein, A., Falkner, S., Bartels, S., Hennig, P., Hutter, F.: Fast Bayesian optimization of machine learning hyperparameters on large datasets. In: Artificial Intelligence and Statistics, pp. 528–536. PMLR (2017)
Knowles, J.: Parego: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Trans. Evol. Comput. 10(1), 50–66 (2006)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: a novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. 18(1), 6765–6816 (2017)
Li, Y., Shen, Y., Jiang, J., Gao, J., Zhang, C., Cui, B.: MFES-HB: efficient hyperband with multi-fidelity quality measurements. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 8491–8500 (2021)
Loshchilov, I., Hutter, F.: CMA-ES for hyperparameter optimization of deep neural networks. arXiv preprint arXiv:1604.07269 (2016)
Nilsback, M.E., Zisserman, A.: A visual vocabulary for flower classification. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 2, pp. 1447–1454. IEEE (2006)
Ozaki, Y., Tanigaki, Y., Watanabe, S., Onishi, M.: Multiobjective tree-structured Parzen estimator for computationally expensive optimization problems. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 533–541 (2020)
Picheny, V.: Multiobjective optimization using gaussian process emulators via stepwise uncertainty reduction. Stat. Comput. 25(6), 1265–1280 (2015)
Ponweiser, W., Wagner, T., Biermann, D., Vincze, M.: Multiobjective optimization on a limited budget of evaluations using model-assisted \(\cal{S}\)-metric selection. In: Rudolph, G., Jansen, T., Beume, N., Lucas, S., Poloni, C. (eds.) PPSN 2008. LNCS, vol. 5199, pp. 784–794. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-87700-4_78
Sener, O., Koltun, V.: Multi-task learning as multi-objective optimization. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 525–536 (2018)
Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:1906.02243 (2019)
Suzuki, S., Takeno, S., Tamura, T., Shitara, K., Karasuyama, M.: Multi-objective Bayesian optimization using pareto-frontier entropy. In: International Conference on Machine Learning, pp. 9279–9288. PMLR (2020)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
Zuluaga, M., Sergent, G., Krause, A., Püschel, M.: Active learning for multi-objective optimization. In: International Conference on Machine Learning, pp. 462–470. PMLR (2013)
Acknowledgements
This work is supported by the Fundamental Research Funds for the Central Universities, Sun Yat-sen University (22qntd1101). It is also supported by the National Natural Science Foundation of China (62162063, 61703183), Science and Technology Planning Project of Guangxi (2021AC19308), and Zhejiang Province Public Welfare Technology Application Research Project of China (LGG19F030010).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, Z., Zhou, Y., Huang, Z., Xia, X. (2022). Towards Efficient Multiobjective Hyperparameter Optimization: A Multiobjective Multi-fidelity Bayesian Optimization and Hyperband Algorithm. In: Rudolph, G., Kononova, A.V., Aguirre, H., Kerschke, P., Ochoa, G., Tušar, T. (eds) Parallel Problem Solving from Nature – PPSN XVII. PPSN 2022. Lecture Notes in Computer Science, vol 13398. Springer, Cham. https://doi.org/10.1007/978-3-031-14714-2_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-14714-2_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14713-5
Online ISBN: 978-3-031-14714-2
eBook Packages: Computer ScienceComputer Science (R0)