Kernel ensemble support vector machine with integrated loss in shared parameters space
Authors: 
YuRen Wu, Xiang-Jun Shen, Stanley Ebhohimhen Abhadiomhen, Yang Yang, Ji-Nan GuAuthors Info & Claims
Pages 18077 - 18096
Abstract
In this paper, we propose a kernel ensemble SVM with integrated loss in shared parameters space. Different from the traditional multiple kernel learning methods of seeking linear combinations of basis kernels as a unified kernel, the proposed method aims to find multiple solutions in corresponding Reproducing Kernel Hilbert Spaces (RKHSs) simultaneously. To achieve this goal, we draw on the idea of multi-view data processing, and the individual kernel gram matrix is considered as one view of the data. We, therefore, propose an ensemble idea to combine those multiple individual kernel losses into a whole one through an integrated loss design. Therefore, each model can co-optimize to learn its optimal parameters by minimizing the integrated loss in multiple RKHSs. Besides, another feature of our method is the introduction of shared and specific parameters in multiple RKHSs for learning. In this manner, the proposed model can learn the common and individual structures of the data from its parameters space, thereby improving the accuracy of the classification task and further enhancing the robustness of the proposed ensemble model. Experimental results on several UCI classification and image datasets demonstrate that our method performs best among state-of-the-art MKL methods, such as SimpleMKL, EasyMKL, MREKLM, and MRMKL.
References
[1]
Aiolli F and Donini M Easymkl: a scalable multiple kernel learning algorithm Neurocomputing 2015 169 215-224
[2]
Anstreicher KM On convex relaxations for quadratically constrained quadratic programming Math Program 2012 136 2 233-251
[3]
Bach FR, Lanckriet GR, Jordan MI (2004) Multiple kernel learning, conic duality, and the smo algorithm. In: Proceedings of the twenty-first international conference on machine learning, p 6
[4]
Bach FR, Thibaux R, and Jordan MI Computing regularization paths for learning multiple kernels Adv Neural Inf Process Syst 2005 17 73-80
[5]
Cortes C, Mohri M, and Rostamizadeh A Algorithms for learning kernels based on centered alignment J Machine Learn Res 2012 13 1 795-828
[6]
Crammer K, Keshet J, and Singer Y Kernel design using boosting Adv Neural Inf Process Syst 2002 15 553-560
[7]
Cristianini N, Shawe-Taylor J, et al. An introduction to support vector machines and other kernel-based learning methods 2000 Cambridge Cambridge University Press
[8]
Fan Q, Wang Z, Zha H, and Gao D Mreklm: a fast multiple empirical kernel learning machine Pattern Recogn 2017 61 197-209
[9]
Gönen M and Alpaydın E Multiple kernel learning algorithms J Machine Learn Res 2011 12 2211-2268
[10]
Gönen M, Khan S, Kaski S (2013) Kernelized bayesian matrix factorization. In: International conference on machine learning, PMLR, pp 864–872
[11]
Han Y, Yang Y, Li X, Liu Q, and Ma Y Matrix-regularized multiple kernel learning via (r,p) norms IEEE Trans Neur Netw Learn Syst 2018 29 10 4997-5007
[12]
Han Y, Yang K, Yang Y, and Ma Y Localized multiple kernel learning with dynamical clustering and matrix regularization IEEE Trans Neural Netw Learn Systs 2016 29 2 486-499
[13]
Hinrichs C, Singh V, Peng J, and Johnson S Q-mkl: Matrix-induced regularization in multi-kernel learning with applications to neuroimaging Adv Neural Inf Process Syst 2012 25 1421-1429
[14]
Hu M, Chen Y, and Kwok JT-Y Building sparse multiple-kernel svm classifiers IEEE Trans Neur Netw 2009 20 5 827-839
[15]
Kim TK T test as a parametric statistic Korean J Anesthesiol 2015 68 6 540
[16]
Kimeldorf G and Wahba G Some results on tchebycheffian spline functions J Math Anal Appl 1971 33 1 82-95
[17]
Kloft M, Brefeld U, Sonnenburg S, and Zien A Lp-norm multiple kernel learning J Machine Learn Res 2011 12 953-997
[18]
Kwok J-Y and Tsang I-H The pre-image problem in kernel methods IEEE Trans Neural Networks 2004 15 6 1517-1525
[19]
Lanckriet GR, Cristianini N, Bartlett P, Ghaoui LE, and Jordan MI Learning the kernel matrix with semidefinite programming J Mach Learn Res 2004 5 Jan 27-72
[20]
Lanckriet GR, De Bie T, Cristianini N, Jordan MI, and Noble WS A statistical framework for genomic data fusion Bioinformatics 2004 20 16 2626-2635
[21]
Li Z, Jing X-Y, Zhu X, Zhang H (2017) Heterogeneous defect prediction through multiple kernel learning and ensemble learning. In: 2017 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 91–102
[22]
Li J, Li Z, Lu G, Xu Y, Zhang B, and Zhang D Asymmetric gaussian process multi-view learning for visual classification Inf Fusion 2021 65 108-118
[23]
Liu J, Chen Y, Zhang J, and Xu Z Enhancing low-rank subspace clustering by manifold regularization IEEE Trans Image Process 2014 23 9 4022-4030
[24]
Liu H, Wang W, Jiao P (2019) Content based image retrieval via sparse representation and feature fusion. In: 2019 IEEE 8th data driven control and learning systems conference (DDCLS). IEEE, pp 18–23
[25]
Mao Q, Tsang IW, Gao S, and Wang L Generalized multiple kernel learning with data-dependent priors IEEE Transactions on Neural Networks and Learning Systems 2014 26 6 1134-1148
[26]
Margineantu DD, Dietterich TG (1997) Pruning adaptive boosting. In: ICML, vol 97, Citeseer, pp 211–218
[27]
Ong CS, Smola A, Williamson R et al (2022) Learning the kernel with hyperkernels
[28]
Qi C, Zhou Z, Hu L, Wang Q (2016) A framework of multiple kernel ensemble learning for hyperspectral classification. In: 2016 intl IEEE conferences on ubiquitous intelligence & computing, advanced and trusted computing, scalable computing and communications, cloud and big data computing, internet of people, and smart world congress (UIC/ATC/ScalCom/CBDCom/IoP/SmartWorld). IEEE, pp 456–460
[29]
Rakotomamonjy A, Bach F, Canu S, and Grandvalet Y Simplemkl J Machine Learn Res 2008 9 2491-2521
[30]
Recht B, Roelofs R, Schmidt L, Shankar V (2022) Do cifar-10 classifiers generalize to cifar-10? arXiv:1806.00451
[31]
Scholkopf B, Smola AJ (2001) Learning with kernels: Support vector machines, regularization, optimization and beyond
[32]
Shen X-J, Ni C, Wang L, and Zha Z-J Sliker: Sparse loss induced kernel ensemble regression Pattern Recogn 2021 109 107587
[33]
Sonnenburg S, Rätsch G, and Schäfer C A general and efficient multiple kernel learning algorithm Adv Neural Inf Process Syst 2005 18 1273-1280
[34]
Sonnenburg S, Rätsch G, Schäfer C, and Schölkopf B Large scale multiple kernel learning J Machine Learn Res 2006 7 1531-1565
[35]
Sun T, Jiao L, Liu F, Wang S, and Feng J Selective multiple kernel learning for classification with ensemble strategy Pattern Recogn 2013 46 11 3081-3090
[36]
Sun L, Zhao C, Yan Z, Liu P, Duckett T, and Stolkin R A novel weakly-supervised approach for rgb-d-based nuclear waste object detection IEEE Sensors J 2018 19 9 3487-3500
[37]
Varma M, Babu BR (2009) More generality in efficient multiple kernel learning. In: Proceedings of the 26th annual international conference on machine learning, pp 1065–1072
[38]
Vishwanathan S, Murty MN (2002) Ssvm: a simple svm algorithm. In: Proceedings of the 2002 international joint conference on neural networks. IJCNN’02 (Cat. No. 02CH37290), vol 3. IEEE, pp 2393–2398
[39]
Wang Q, Ding Z, Tao Z, Gao Q, Fu Y (2020) Generative partial multi-view clustering. arXiv:2003.13088
[40]
Wu D, Wang B, Precup D, and Boulet B Multiple kernel learning-based transfer regression for electric load forecasting IEEE Transactions on Smart Grid 2019 11 2 1183-1192
[41]
Xia H and Hoi SC Mkboost: a framework of multiple kernel boosting IEEE Trans Knowl Data Eng 2012 25 7 1574-1586
[42]
Xu X, Tsang IW, and Xu D Soft margin multiple kernel learning IEEE Trans Neur Netw Learn Syst 2013 24 5 749-761
[43]
Ye J, Ji S, Chen J (2020) Multi-class discriminant kernel learning via convex programming. Journal of Machine Learning Research 9(4)
[44]
Zheng-Peng W and Zhang X-G Elastic multiple kernel learning Acta Automatica Sinica 2011 37 6 693-699
[45]
Zhou J, Jiang Z, Chung F-L, Wang S (2022) Formulating ensemble learning of svms into a single svm formulation by negative agreement learning. IEEE Transactions on Systems Man and Cybernetics Systems
[46]
Zhou J, Liu J, Luo X (2015) Rademacher complexity bound for domain adaptation regression. In: 2015 conference on technologies and applications of artificial intelligence (TAAI). IEEE, pp 273–280
Recommendations
Multi-view Representation Induced Kernel Ensemble Support Vector Machine
AbstractThis paper proposes a multi-view representation kernel ensemble Support Vector Machine. Unlike the conventional multiple kernel learning techniques which utilizes a common similarity measure over the entire input space, with the aim of solely ...
Robust kernel ensemble regression in diversified kernel space with shared parameters
AbstractKernel regression is an effective non-parametric regression method. However, such regression methods have problems in choosing an appropriate kernel and its parameters. In this paper, we propose a robust kernel ensemble regression model (RKER) in ...
MKEL: Multiple Kernel Ensemble Learning via Unified Ensemble Loss for Image Classification
In this article, a novel ensemble model, called Multiple Kernel Ensemble Learning (MKEL), is developed by introducing a unified ensemble loss. Different from the previous multiple kernel learning (MKL) methods, which attempt to seek a linear combination ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 01 December 2022
Accepted: 04 November 2022
Revision received: 18 February 2022
Received: 20 September 2021
Author Tags
Qualifiers
- Research-article
Funding Sources
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 24 Sep 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in