research-article

Robust Lp-norm least squares support vector regression with feature selection

Authors:

Xiang-Yu HuaAuthors Info & Claims

Applied Mathematics and Computation, Volume 305, Issue C

Pages 32 - 52

https://doi.org/10.1016/j.amc.2017.01.062

Published: 15 July 2017 Publication History

Abstract

Lp-norm least squares support vector regression (Lp-LSSVR) is proposed for feature selection in regression.Using the absolute constraint and the Lp-norm regularization term, Lp-LSSVR performs robust against outliers.Lp-LSSVR ensures the useful features to be selected based on theoretical analysis.Lp-LSSVR only solves a series of linear equations, leading to fast training speed. In this paper, we aim a novel algorithm called robust Lp-norm least squares support vector regression (Lp-LSSVR) that is more robust than the traditional least squares support vector regression(LS-SVR). Using the absolute constraint and the Lp-norm regularization term, our Lp-LSSVR performs robust against outliers. Moreover, though the optimization problem is non-convex, the sparse solution of Lp-norm and the lower bonds for nonzero components technique ensure useful features selected by Lp-LSSVR, and it helps to find the local optimum of our Lp-LSSVR. Experimental results show that although Lp-LSSVR is more robust than least squares support vector regression (LS-SVR), and much faster than Lp-norm support vector regression (Lp-SVR) and SVR due to its equality constraint, it is slower than LS-SVR and L1-norm support vector regression (L1-SVR), it is as effective as Lp-SVR, L1-SVR, LS-SVR and SVR in both feature selection and regression.

References

[1]

H. Drucker, C.J.C. Burges, L. Kaufman, A. Smola, V. Vapnik, Support Vector Regression Machines, Bell Labs and Monmouth University, 1997.

[2]

C. Cortes, V.N. Vapnik, Support vector networks, Mach. Learn., 20 (1995) 273-297.

Digital Library

[3]

V. Vapnik, Statistical Learning Theory, Wiley, New York, 1998.

[4]

C. Burges, A tutorial on support vector machines for pattern recognition, Data Min. Knowl. Discov., 2 (1998) 121-167.

Digital Library

[5]

J. Gu, M. Zhu, L. Jiang, Housing price forecasting based on genetic algorithm and support vector machine, Expert. Syst. Appl., 38 (2011) 3383-3386.

Digital Library

[6]

Z. Chen, Z. Fan, Dynamic customer lifetime value prediction using longitudinal data: an improved multiple kernel SVR approach, Expert. Syst. Appl., 43 (2013) 123-134.

Digital Library

[7]

S. Huang, P. Chuang, C. Wu, H. Lai, Chaos-based support vector regressions for exchange rate forecasting, Expert. Syst. Appl., 37 (2010) 8590-8598.

Digital Library

[8]

Y. Wang, H. Ni, S. Wang, Nonparametric bivariate copula estimation based on shape-restricted support vector regression, Knowl-Based Syst., 35 (2012) 235-244.

Digital Library

[9]

X. Tang, L. Zhuang, C. Jiang, Prediction of silicon content in hot metal using support vector regression based on chaos particle swarm optimization, Expert. Syst. Appl., 36 (2009) 11853-11857.

Digital Library

[10]

J. Huang, Y. Bo, H. Wang, Electromechanical equipment state forecasting based on genetic algorithm support vector regression, Expert. Syst. Appl., 38 (2011) 8399-8402.

Digital Library

[11]

R. Kohavi, G. John, Wrappers for feature subset slection, Artif. Intell., 97 (1997) 273-324.

Digital Library

[12]

I. Guyon, A. Elisseeff, An introduction to variable and feature selection, J. Mach. Learn. Res., 3 (2003) 1157-1182.

Digital Library

[13]

C. Huang, C. Tsai, A hybrid SOFM-SVR with a filter-based feature selection for stock market forecasting, Expert. Syst. Appl., 36 (2009) 1529-1539.

Digital Library

[14]

J. Yang, C. Ong, Feature selection using probabilistic prediction of support vector regression, IEEE T. Neural. Networ., 22 (2011) 954-962.

Digital Library

[15]

C. Zhang, D. Li, J. Tan, The support vector regression with adaptive norms, Procedia Comput. Sci., 18 (2013) 1730-1736.

[16]

R. Tibshirani, Regression shrinkage and selection via the lasso, J. R. Stat. Soc. Ser. B, 58 (1996) 207-288.

[17]

Y. Ye, Y. Jiang, Y. Shao, C. Li, Financial conditions index construction through weighted Lp-norm support vector regression, J. Adv. Comput. Intell. Intell. Inf., 19 (2015) 397-406.

[18]

Y. Ye, Y. Shao, C. Li, Wavelet Lp-norm support vector regression with feature selection, J. Adv. Comput. Intell. Intell. Inf., 19 (2015) 407-416.

[19]

X. Peng, D. Xu, A local information-based feature-selection algorithm for data regression, Pattern. Recognit., 46 (2013) 2519-2530.

Digital Library

[20]

I. Guyon, A. Elisseeff, An introduction to variable and feature selection, J. Mach. Learn. Res., 3 (2003) 1157-1182.

Digital Library

[21]

G. Doquire, M. Verleysen, An introduction to variable and feature selection, Neurocomputing, 121 (2013) 5-13.

Digital Library

[22]

N. Deng, Y. Tian, C. Zhang, Support Vector Machines: Optimization-Based Theory, Algorithms, and Extensions, CRC Press, 2013.

Digital Library

[23]

O. Mangasarian, Minimum-support Solutions of Polyhedral Concave Programs Taylor & Francis, 1997.

[24]

P. Bradley, O. Mangasarian, Feature selection via concave minimization and support vector machines, 1998.

[25]

J. Zhu, S. Rosset, T. Hastie, R. Tibshirani, 1-norm support vector machines, MIT Press, 2003.

[26]

M. Tanveer, Robust and sparse linear programming twin support vector machines, Cogn. Comput. (2014).

[27]

M. Yuan, Y. Lin, Model selection and estimation in regression with grouped variables, J. R. Stat. Soc. Ser. B, 68 (2006) 49-67.

[28]

L. Meier, S. van de Geer, P. Bhlmann, The group lasso for logistic regression, J. R. Stat. Soc. Ser. B, 70 (2008) 53-71.

[29]

J. Suykens, L. Lukas, P. Van Dooren, B. De Moor, J. Vandewalle, Least squares support vector machine classifiers: a large scale algorithm, 1999.

[30]

P. Bradley, O. Mangasarian, W. Street, Feature selection via mathematical programming, INFORMS J. Comput., 10 (1998) 209-217.

Digital Library

[31]

O. Mangasarian, Solution of general linear complementarity problems via nondifferentiable concave minimization, Acta Mathematica Vietnamica, 22 (1997) 199-205.

[32]

O. Mangasarian, Absolute value equation solution via concave minimization, Optim. Lett., 1 (2007) 3-8.

[33]

O. Mangasarian, Absolute value programming, Comput. Optim. Appl., 36 (2007) 43-53.

Digital Library

[34]

J. Tan, Z. Zhang, L. Zhen, C. Zhang, N. Deng, Adaptive feature selection via a new version of support vector machine, Neural Comput. Appl., 23 (2013) 937-945.

[35]

J. Tan, C. Zhang, N. Deng, Cancer related gene identification via p-norm support vector machine, 2010.

[36]

C. Zhang, J. Tan, N. Deng, Feature selection in multiinstance learning, Chengdu, China, 2010.

[37]

W. Chen, Y. Tian, Lp-norm proximal support vector machine and its application, Proc. Comput. Sci., 1 (2010) 2411-2417.

[38]

C. Zhang, Y. Shao, J. Tan, N. Deng, Mixed-norm linear support vector machine, Neural Comput. Appl., 23 (2013) 2159-2166.

[39]

J. Suykens, J. Brabanter, L. Lukas, J. Vandewalle, Weighted least squares support vector machines: robustness and sparse approximation, Neurocomputing, 48 (2002) 85-105.

[40]

R. Mall, J. Suykens, Very sparse LSSVM reductions for large scale data, IEEE T. Neur. Net. Lear., 26 (2015) 1086-1097.

[41]

Y. Shao, C. Zhang, Z. Yang, L. Jing, N. Deng, An -twin support vector machine for regression, Neural Comput. Appl., 23 (2008) 175-185.

[42]

G. Sierksma, Linear and Integer Programming: Theory and Practice, CRC Press, 2001.

[43]

X. Chen, F. Xu, Y. Ye, Lower bound theory of nonzero entries in solutions of l2lp minimization, SIAM J. Sci. Comput., 32 (2009) 2832-2852.

[44]

O. Mangasarian, E. Wild, Feature selection for nonlinear kernel support vector machines, 2007.

[45]

S. Keerthi, C. Lin, Asymptotic behaviors of support vector machines with gaussian kernel, Neural Comput., 15 (2003) 1667-1689.

Digital Library

[46]

G.B. Huang, Q.Y. Zhu, C.K. Siew, Extreme learning machine: theory and applications, Neurocomputing, 70 (2006) 489-501.

[47]

Y. Ye, H. Cao, L. Bai, Z. Wang, Y. Shao, Exploring determinants of inflation in china based on l1- twin support vector regression, Proc. Comput. Sci., 17 (2013) 514-522.

[48]

Y. Ye, Y. Shao, W. Chen, Comparing inflation forecasts using an -wavelet twin support vector regression, J. Inf. Comput. Sci., 10 (2013) 2041-2049.

[49]

Z. Yang, X. Hua, Y. Shao, Y. Ye, A novel parametric-insensitive nonparallel support vector machine for regression, Neurocomputing, 171 (2016) 649-663.

Digital Library

[50]

W. Cui, X. Yan, Adaptive weighted least square support vector machine regression integrated with outlier detection and its application in QSAR, Chenometrics Intell. Lab. Syst., 98 (2009) 130-135.

[51]

P. Hao, Interval regression analysis using support vector networks, Fuzzy Set. Syst., 160 (2009) 2466-2485.

Digital Library

[52]

R. Eubank, Nonparametric Regression and Spline Smoothing Statistics: Textbooks and Monographs, Marcel Dekker, New York, 1999.

[53]

W. Wen, Z. Hao, X. Yang, Robust least squares support vector machine based on recursive outlier elimination, Soft. Comput., 14 (2010) 1241-1251.

Digital Library

[54]

J. Stock, M. Watson, Forecasting output and inflation:the role of asset prices, J. Econ. Lit., 41 (2003) 788-829.

[55]

Q. Yu, Y. Miche, E. Sverin, Bankruptcy prediction using extreme learning machine and financial expertise, Neurocomputing, 128 (2014) 296-302.

Digital Library

[56]

S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Found. Trends Mach. Learn., 3 (2010) 1-122.

Digital Library

[57]

Y. Feng, S. Lv, H. Hang, J. Suykens, Kernelized elastic net regularization: generalization bounds and sparse recovery, Neural Comput., 28 (2016) 525-562.

Digital Library

[58]

S. Salzo, J. Suykens, Generalized support vector regression: duality and tensor-kernel representation, arXiv:1603.05876v1 {math.OC} (2016).

Cited By

Yang MLiang HWu XZhang Z(2025)A flexible and efficient algorithm for high dimensional support vector regressionNeurocomputing10.1016/j.neucom.2024.128671611:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.neucom.2024.128671
Xu KXu YYe YChen W(2022)Novel Feature Selection Method for Nonlinear Support Vector RegressionComplexity10.1155/2022/47401732022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/4740173
Wu XChen HLi TWan J(2021)Semi-supervised feature selection with minimal redundancy based on local adaptiveApplied Intelligence10.1007/s10489-021-02288-451:11(8542-8563)Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1007/s10489-021-02288-4
Show More Cited By

Robust Lp-norm least squares support vector regression with feature selection
1. Computing methodologies

Recommendations

Sparse least square twin support vector machine with adaptive norm

By promoting the parallel hyperplanes to non-parallel ones in SVM, twin support vector machines (TWSVM) have attracted more attention. There are many modifications of them. However, most of the modifications minimize the loss function subject to the I ₂-...
Twin least squares support vector regression

In this paper, combining the spirit of twin hyperplanes with the fast speed of least squares support vector regression (LSSVR) yields a new regressor, termed as twin least squares support vector regression (TLSSVR). As a result, TLSSVR outperforms ...
Sparse Lq-norm least squares support vector machine with feature selection

We propose an Lq-norm LS-SVM with feature selection for small size samples.Feature selection is achieved effectively by minimizing the Lq-norm of weight.The number of selected features can be adjusted by choosing the parameters.An efficient iterative ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Applied Mathematics and Computation

Applied Mathematics and Computation Volume 305, Issue C

July 2017

309 pages

ISSN:0096-3003

Issue’s Table of Contents

Copyright © Elsevier Inc.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 15 July 2017

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang MLiang HWu XZhang Z(2025)A flexible and efficient algorithm for high dimensional support vector regressionNeurocomputing10.1016/j.neucom.2024.128671611:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.neucom.2024.128671
Xu KXu YYe YChen W(2022)Novel Feature Selection Method for Nonlinear Support Vector RegressionComplexity10.1155/2022/47401732022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/4740173
Wu XChen HLi TWan J(2021)Semi-supervised feature selection with minimal redundancy based on local adaptiveApplied Intelligence10.1007/s10489-021-02288-451:11(8542-8563)Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1007/s10489-021-02288-4
Guijo-Rubio DVargas VGutiérrez PHervás-Martínez C(2021)Studying the Effect of Different Norms in the Context of Time Series Ordinal ClassificationAdvances in Artificial Intelligence10.1007/978-3-030-85713-4_5(44-53)Online publication date: 22-Sep-2021
https://dl.acm.org/doi/10.1007/978-3-030-85713-4_5
Singla MShukla K(2020)Robust statistics-based support vector machine and its variants: a surveyNeural Computing and Applications10.1007/s00521-019-04627-632:15(11173-11194)Online publication date: 1-Aug-2020
https://dl.acm.org/doi/10.1007/s00521-019-04627-6
Li CShao YZhao DGuo YHua X(2020)Feature selection for high‐dimensional regression via sparse LSSVR based on Lp‐normInternational Journal of Intelligent Systems10.1002/int.2233436:2(1108-1130)Online publication date: 24-Nov-2020
https://dl.acm.org/doi/10.1002/int.22334
Shi YMiao JNiu L(2019)Feature selection with MCP$$^2$$ regularizationNeural Computing and Applications10.1007/s00521-018-3500-731:10(6699-6709)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1007/s00521-018-3500-7
Dong YJiang H(2018)A Two-Stage Regularization Method for Variable Selection and Forecasting in High-Order Interaction ModelComplexity10.1155/2018/20329872018Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1155/2018/2032987

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents