Abstract
This paper investigates the impact of kernel functions on the accuracy of bi-fidelity Gaussian process regressions (GPR) for engineering applications. The potential of composite kernel learning (CKL) and model selection is also studied, aiming to ease the process of manual kernel selection. Using the autoregressive Gaussian process as the base model, this paper studies four kernel functions and their combinations: Gaussian, Matern-3/2, Matern-5/2, and Cubic. Experiments on four engineering test problems show that the best kernel is problem dependent and sometimes might be counter-intuitive, even when a large amount of low-fidelity data already aids the model. In this regard, using CKL or automatic kernel selection via cross validation and maximum likelihood can reduce the tendency to select a poor-performing kernel. In addition, the CKL technique can create a slightly more accurate model than the best-performing individual kernel. The main drawback of CKL is its significantly expensive computational cost. The results also show that, given a sufficient amount of samples, tuning the regression term is important to improve the accuracy and robustness of bi-fidelity GPR, while decreasing the importance of the proper kernel selection.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bachoc F (2013) Cross validation and maximum likelihood estimations of hyper-parameters of Gaussian processes with model misspecification. Comput Stat Data Anal 66:55–69
Bertram A, Othmer C, Zimmermann R (2018) Towards real-time vehicle aerodynamic design via multi-fidelity data-driven reduced order modeling. In: 2018 AIAA/ASCE/AHS/ASC structures. structural dynamics, and materials conference 0916
Bonfiglio, L., Perdikaris P, Brizzolara S, Karniadakis G (2017) A multi-fidelity framework for investigating the performance of super-cavitating hydrofoils under uncertain flow conditions. In: 19th AIAA non-deterministic approaches conference, p 1328
Bostanabad R, Kearney T, Tao S, Apley DW, Chen W (2018) Leveraging the nugget parameter for efficient Gaussian process modeling. Int J Numer Methods Eng 114(5):501–516
Brevault L, Balesdent M, Hebbal A (2020) Overview of Gaussian process based multi-fidelity techniques with variable relationship between fidelities, application to aerospace systems. Aerosp Sci Technol 107:106339
Bryson DE, Rumpfkeil MP (2017) All-at-once approach to multifidelity polynomial chaos expansion surrogate modeling. Aerosp Sci Technol 70:121–136
Cère-Aéro (2022) Flow5 v7.15—documentation. https://flow5.tech/
Cutajar K, Pullin M, Damianou A, Lawrence N, González J (2019) Deep Gaussian processes for multi-fidelity modeling. arXiv preprint. arXiv:1903.07320
de Baar J, Roberts S, Dwight R, Mallol B (2015) Uncertainty quantification for a sailing yacht hull, using multi-fidelity Kriging. Comput Fluids 123:185–201
Economon TD, Palacios F, Copeland SR, Lukaczyk TW, Alonso JJ (2016) Su2: an open-source suite for multiphysics simulation and design. AIAA J 54(3):828–846
Fernández-Godino MG, Park C, Kim NH, Haftka RT (2016) Review of multi-fidelity models. arXiv preprint. arXiv:1609.07196
Han Z-H, Görtz S (2012) Hierarchical Kriging model for variable-fidelity surrogate modeling. AIAA J 50(9):1885–1896
Isogai K (1979) On the transonic-dip mechanism of flutter of a sweptback wing. AIAA J 17(7):793–795
Jin S-S (2020) Compositional kernel learning using tree-based genetic programming for Gaussian process regression. Struct Multidisc Optim 62:1313–1351
Jofre L, Geraci G, Fairbanks H, Doostan A, Iaccarino G (2018) Multi-fidelity uncertainty quantification of irradiated particle-laden turbulence. arXiv preprint. arXiv:1801.06062
Kennedy MC, O’Hagan A (2000) Predicting the output from a complex computer code when fast approximations are available. Biometrika 87(1):1–13
Kersaudy P, Sudret B, Varsier N, Picon O, Wiart J (2015) A new surrogate modeling technique combining Kriging and polynomial chaos expansions—application to uncertainty analysis in computational dosimetry. J Comput Phys 286:103–117
Konakli K, Sudret B (2016) Reliability analysis of high-dimensional models using low-rank tensor approximations. Probab Eng Mech 46:18–36
Kronberger G, Kommenda M (2013) Evolution of covariance functions for Gaussian process regression using genetic programming. In: Moreno-Díaz R, Pichler F, Quesada-Arencibia A (eds) Computer aided systems theory—EUROCAST 2013. Springer, Berlin, pp 308–315
Le Gratiet L, Garnier J (2012) Recursive co-Kriging model for design of computer experiments with multiple levels of fidelity. Int J Uncertain Quant 4(5):365–386
Liu B, Koziel S, Zhang Q (2016) A multi-fidelity surrogate-model-assisted evolutionary algorithm for computationally expensive optimization problems. J Comput Sci 12:28–37
Liu X, Zhao W, Wan D (2022) Multi-fidelity co-Kriging surrogate model for ship hull form optimization. Ocean Eng 243:110239
Maolin S, Liye L, Sun W, Xueguan S (2020) A multi-fidelity surrogate model based on support vector regression. Struct Multidisc Optim 61(6):2363–2375
Meng X, Wang Z, Fan D, Triantafyllou MS, Karniadakis GE (2021) A fast multi-fidelity method with uncertainty quantification for complex data correlations: application to vortex-induced vibrations of marine risers. Comput Methods Appl Mech Eng 386:114212
Ng LWT, Eldred M (2012) Multifidelity uncertainty quantification using non-intrusive polynomial chaos and stochastic collocation. In: 53rd AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics and materials conference, 20th AIAA/ASME/AHS adaptive structures conference, 14th AIAA, p 1852
Palar PS, Shimoyama K (2018) On efficient global optimization via universal Kriging surrogate models. Struct Multidisc Optim 57(6):2377–2397
Palar PS, Shimoyama K (2019) Efficient global optimization with ensemble and selection of kernel functions for engineering design. Struct Multidisc Optim 59(1):93–116
Palar PS, Tsuchiya T, Parks GT (2016) Multi-fidelity non-intrusive polynomial chaos based on regression. Comput Methods Appl Mech Eng 305:579–606
Palar PS, Zuhal LR, Shimoyama K, Tsuchiya T (2018) Global sensitivity analysis via multi-fidelity polynomial chaos expansion. Reliab Eng Syst Saf 170:175–190
Palar PS, Parussini L, Bregant L, Shimoyama K, Izzaturrahman MF, Baehaqi FA, Zuhal L (2022) Composite kernel functions for surrogate modeling using recursive multi-fidelity Kriging. In: AIAA SCITECH 2022 forum, p 0506
Pang G, Perdikaris P, Cai W, Karniadakis GE (2017) Discovering variable fractional orders of advection–dispersion equations from field data using multi-fidelity bayesian optimization. J Comput Phys 348:694–714
Park C, Haftka RT, Kim NH (2017) Remarks on multi-fidelity surrogates. Struct Multidisc Optim 55(3):1029–1050
Perdikaris P, Raissi M, Damianou A, Lawrence ND, Karniadakis GE (2017) Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling. Proc R Soc A Math Phys Eng Sci 473(2198):20160751
Ranjan P, Haynes R, Karsten R (2011) A computationally stable approach to Gaussian process interpolation of deterministic computer simulation data. Technometrics 53(4):366–378
Satria Palar P, Rizki Zuhal L, Shimoyama K (2020) Gaussian process surrogate model with composite kernel learning for engineering design. AIAA J 58(4):1864–1880
Serani A, Pellegrini R, Wackers J, Jeanson C-E, Queutey P, Visonneau M, Diez M (2019) Adaptive multi-fidelity sampling for CFD-based optimisation via radial basis function metamodels. Int J Comput Fluid Dyn 33(6–7):237–255
Song X, Lv L, Sun W, Zhang J (2019) A radial basis function-based multi-fidelity surrogate model: exploring correlation between high-fidelity and low-fidelity models. Struct Multidisc Optim 60(3):965–981
Sundar V, Shields MD (2019) Reliability analysis using adaptive kriging surrogates with multimodel inference. ASCE-ASME J Risk Uncertain Eng Syst Part A Civ Eng 5(2):04019004
Tao J, Sun G (2019) Application of deep learning based multi-fidelity surrogate model to robust aerodynamic design optimization. Aerosp Sci Technol 92:722–737
Toal DJ (2015) Some considerations regarding the use of multi-fidelity Kriging in the construction of surrogate models. Struct Multidisc Optim 51(6):1223–1245
Yoo K, Bacarreza O, Aliabadi MF (2020) A novel multi-fidelity modelling-based framework for reliability-based design optimisation of composite structures. Eng Comput 38:595–608
Zhang X, Xie F, Ji T, Zhu Z, Zheng Y (2021) Multi-fidelity deep neural network surrogate model for aerodynamic shape optimization. Comput Methods Appl Mech Eng 373:113485
Zuhal LR., Faza GA, Palar PS, Liem RP (2021) On dimensionality reduction via partial least squares for kriging-based reliability analysis with active learning. Reliab Eng Syst Saf 215:107848
Acknowledgements
The authors acknowledge financial support from Penelitian Dasar Unggulan Perguruan Tinggi research scheme administered by Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi, Republic of Indonesia.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Replications of results
The code and the data needed to replicate the results can be downloaded from the following link: https://github.com/optimuspram/MF-GPR-code.
Additional information
Responsible Editor: Ramin Bostanabad
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1: Comparison to single-fidelity Gaussian process
For the sake of completeness, Table 21 shows the comparison of the NRMSE between the single- and bi-fidelity CKL for all problems and combinations of \(n_{h}\) and \(n_{l}\). This appendix shows that the bi-fidelity model is better than the single-fidelity model for equivalent cost. Consider a combination of low- and high-fidelity data sets in which the cost ratio is defined as
where \(t_{\text {high}}\) and \(t_{\text {low}}\) are the wall-clock time of the high- and low-fidelity simulations, respectively. The equivalent sample size of the single-fidelity model, given \(n_{h}\) and \(n_{l}\), is defined as
In this regard, a single-fidelity GP with \(n_{\text {equiv}}\) samples is then compared with the corresponding bi-fidelity GP with \(n_{h}\) high-fidelity and \(n_{l}\) low-fidelity samples. Notice that there is no \(n_{\text {equiv}}\) for the vibration rig problem since the high-fidelity data are evaluated experimentally. Hence, we use the single-fidelity GP using \(n_{h}\) samples for the vibration rig problem. The averaged NRMSE results for all problems, tuned \(\lambda\) case, are shown in Table 21. It can be seen that the bi-fidelity GP with CKL always outperforms its single-fidelity GP counterparts, with the only exception being on the heat conduction case with \(n_{h}=40/n_{l}=120.\)
Appendix 2: Examples of sample paths from Gaussian processes
To illustrate the behavior of the composite kernels, Figs. 13 and 14 show the sample paths generated from the four individual kernels and CKL with various weights, respectively. The sample paths were generated using the lengthscale of 0.1 for all kernels. As shown in Fig. 14, the sample paths from composite kernels reflect the behavior of the constituents. Let us denote the vector of weight as follows: \(\varvec{w}=[\text {Gaussian, Matern-3/2, Matern-5/2, Cubic}]\). The weights shown in The CKL with \(\varvec{w} = [0.49,0,0,0.51]\) (extracted from the two-variable Isogai problem) is a combination of only the Gaussian and cubic kernel, with the sample paths reflect the smooth nature of Gaussian and the rapid change of the cubic. On the other hand, the combination of Matern-3/2 and cubic kernel (\(\varvec{w}=[0,0.57,0,0.43]\), extracted from the subsonic wing problem) yields a rough characteristic that primarily comes from the former. The combination of Gaussian, Matern-5/2, and cubic (\(\varvec{w}=[0.44,0,0.31,0.25]\), extracted from the eight-variable subsonic wing problem) primarily exhibits the smoothness of Gaussian, with slight roughness from Matern-5/2 and cubic. Finally, the combination of dominant Gaussian kernel and cubic (\(\varvec{w}=[0.97,0,0.03,0]\), extracted from the ten-variable heat conduction problem) almost looks similar to Gaussian; however, the slight change of this characteristic can lead to a better prediction as observed in the ten-variable heat conduction problem.
Examples of 20 sample paths generated from various kernel functions using \(\theta =0.1\)
Examples of 20 sample paths generated from composite kernels with various weights using \(\theta =0.1\). The composition of the weight vector is as follows: \(\varvec{w}=[\text{Gaussian,Matern-3/2,Matern-5/2,Cubic}]\)
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Palar, P.S., Parussini, L., Bregant, L. et al. On kernel functions for bi-fidelity Gaussian process regressions. Struct Multidisc Optim 66, 37 (2023). https://doi.org/10.1007/s00158-023-03487-y
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s00158-023-03487-y