Abstract
Bayesian Optimization has become the reference method for the global optimization of black box, expensive and possibly noisy functions. Bayesian Optimization learns a probabilistic model about the objective function, usually a Gaussian Process, and builds, depending on its mean and variance, an acquisition function whose optimizer yields the new evaluation point, leading to update the probabilistic surrogate model. Despite its sample efficiency, Bayesian Optimization does not scale well with the dimensions of the problem. Moreover, the optimization of the acquisition function has received less attention because its computational cost is usually considered negligible compared to that of the evaluation of the objective function: its efficient optimization is also inhibited, particularly in high dimensional problems, by multiple extrema and “flat” regions. In this paper we leverage the additivity – aka separability – of the objective function into mapping both the kernel and the acquisition function of the Bayesian Optimization in lower dimensional subspaces. This approach makes more efficient both the learning/updating of the probabilistic surrogate model and the optimization of the acquisition function. Experimental results are presented for a standard test function and a real-life application.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Frazier., P.I.: Bayesian optimization. In: INFORMS TutORials in Operations Research, pp. 255–278 (2018)
Archetti, F., Candelieri, A.: Bayesian Optimization and Data Science. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24494-1
Krige, D.G.: A statistical approach to some basic mine valuation problems on the Witwatersrand. J. Chem. Metal. Min. Soc. South Africa 52, 119–139 (1951)
Kushner, H.J.: A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. J. Basic Eng. 86(1), 97–106 (1964)
Močkus, J.: On Bayesian methods for seeking the extremum. In: Marchuk, G.I. (ed.) Optimization Techniques IFIP Technical Conference. LNCS, pp. 400–404. Springer, Heidelberg (1975). https://doi.org/10.1007/978-3-662-38527-2_55
Zhigljavsky, A., Zilinskas, A.: Stochastic Global Optimization, vol. 9. Springer, Boston (2007). https://doi.org/10.1007/978-0-387-74740-8
Gramacy, R.B., Lee, H.K.M., Holmes, C., Osborne, M.: Optimization under unknown constraints. Bayesian Stat. 9, 229 (2012)
Candelieri, A., Archetti, F.: Sequential model based optimization with black-box constraints: feasibility determination via machine learning. In: AIP Conference Proceedings, vol. 2070, no. 1, p. 020010. AIP Publishing LLC, February 2019
Hernández-Lobato, J.M., Gelbart, M.A., Hoffman, M.W., Adams, R.P., Ghahramani, Z.: Predictive entropy search for Bayesian Optimization with unknown constraints. In: Proceedings of the 32nd International Conference on Machine Learning, vol. 37 (2015)
Candelieri, A: Sequential model based optimization of partially defined functions under unknown constraints. J. Global Optim. 1–23 (2019)
Hutter, F., Kotthoff, L., Vanschoren, J.: Automated Machine Learning-Methods, Systems, Challenges. Automated Machine Learning. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05318-5
Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20(55), 1–21 (2019)
Candelieri, A., Archetti, F.: Global optimization in machine learning: the design of a predictive analytics application. Soft. Comput. 23(9), 2969–2977 (2018). https://doi.org/10.1007/s00500-018-3597-8
Wang, Z., Hutter, F., Zoghi, M., Matheson, D., de Feitas, N.: Bayesian optimization in a billion dimensions via random embeddings. J. Artif. Intell. Res. 55, 361–387 (2016)
Duvenaud, D.K., Nickisch, H., Rasmussen, C.E.: Additive Gaussian processes. In: Advances in Neural Information Processing Systems, pp. 226–234 (2011)
Duvenaud, D., Lloyd, J.R., Grosse, R., Tenenbaum, J.B., Ghahramani, Z.: Structure discovery in nonparametric regression through compositional kernel search. arXiv preprint arXiv:1302.4922 (2013)
Wilson, A., Adams, R.: Gaussian process kernels for pattern discovery and extrapolation. In: International Conference on Machine Learning, pp. 1067–1075, February 2013
Mutny, M., Krause, A.: Efficient high dimensional Bayesian optimization with additivity and quadrature Fourier features. In: Advances in Neural Information Processing Systems, pp. 9005–9016 (2018)
Zhigljavsky, A., Žilinskas, A.: Selection of a covariance function for a Gaussian random field aimed for modeling global optimization problems. Optim. Lett. 13(2), 249–259 (2019). https://doi.org/10.1007/s11590-018-1372-5
Sergeyev, Y.D.: An information global optimization algorithm with local tuning. SIAM J. Optim. 5(4), 858–870 (1995)
Srinivas, N., Krause, A., Kakade, S.M., Seeger, M.W.: Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Trans. Inf. Theory 58(5), 3250–3265 (2012)
Žilinskas, A., Calvin, J.: Bi-objective decision making in global optimization based on statistical models. J. Global Optim. 74(4), 599–609 (2018). https://doi.org/10.1007/s10898-018-0622-5
Basu, K., Ghosh, S.: Analysis of Thompson sampling for Gaussian process optimization in the bandit setting. arXiv preprint arXiv:1705.06808 (2017)
Jamil, M., Yang, X.S.: A literature survey of benchmark functions for global optimization problems. arXiv preprint arXiv:1308.4008 (2013)
Candelieri, A., Perego, R., Archetti, F.: Bayesian optimization of pump operations in water distribution systems. J. Global Optim. 71(1), 213–235 (2018). https://doi.org/10.1007/s10898-018-0641-2
Acknowledgments
This study has been partially supported by the Italian project “PerFORM WATER 2030” – programme POR (Programma Operativo Regionale) FESR (Fondo Europeo di Sviluppo Regionale) 2014–2020, innovation call “Accordi per la Ricerca e l’Innovazione” (“Agreements for Research and Innovation”) of Regione Lombardia, (DGR N. 5245/2016 - AZIONE I.1.B.1.3 – ASSE I POR FESR 2014–2020) – CUP E46D17000120009.
We greatly acknowledge the DEMS Data Science Lab for supporting this work by providing computational resources.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Candelieri, A., Giordani, I., Perego, R., Archetti, F. (2020). Composition of Kernel and Acquisition Functions for High Dimensional Bayesian Optimization. In: Kotsireas, I., Pardalos, P. (eds) Learning and Intelligent Optimization. LION 2020. Lecture Notes in Computer Science(), vol 12096. Springer, Cham. https://doi.org/10.1007/978-3-030-53552-0_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-53552-0_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-53551-3
Online ISBN: 978-3-030-53552-0
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)