A feature selection approach based on sensitivity of RBFNNs
Abstract
Feature selection is an important issue in pattern recognition and machine learning, which aims at selecting relevant features from a set of candidates. Obviously, the establishment of proper criteria to evaluate the relevance of features is pivotal to the selection. In this paper, a criterion is proposed to assess the relevance of individual input features based on Radial Basis Function Neural Networks (RBFNNs) involved in solving classification and regression problems. The criterion takes a quantified output sensitivity of RBFNNs to input variation as a measure, which is defined as a mathematical expectation and can in a statistic sense reflect the effect of an RBFNN's input variation on its output. The basic idea is that a well-trained RBFNN can capture relevant features of the problem it deals with and thus become more sensitive to the variation of those input features that make more contributions to the RBFNN's behavior. Since the sensitivity is difficult to exactly compute, a numerical integral technique is employed to approximately compute the sensitivity. Experimental results on several artificial and real datasets show that our proposed feature selection approach works well.
References
[1]
L. Song, A. Smola, A. Gretton, J. Bedo, K. Borgwardt, Feature selection via dependence maximization, J. Mach. Learn. Res., 13 (2012) 1393-1434.
[2]
L. Wolf, A. Shashua, Feature selection for unsupervised and supervised inference: the emergence of sparsity in a weight-based approach, J. Mach. Learn. Res., 6 (2005) 1855-1887.
[3]
H. Liu, L. Yu, Toward integrating feature selection algorithms for classification and clustering, IEEE Trans. Knowl. Data Eng., 17 (2005) 491-502.
[4]
X. He, D. Cai, P. Niyogi, Laplacian score for feature selection, Adv. Neural Inf. Process. Syst., 18 (2005) 507-514.
[5]
C. Boutsidis, M. Mahoney, P. Drineas, Unsupervised feature selection for principal components analysis, in: Proceeding of the 14th ACMSIGKDD International Conference on Knowledge Discovery and Data Mining, 2008, pp. 61-69.
[6]
R. Kohavi, G. John, Wrappers for feature subset selection, Artif. Intell., 97 (1997) 273-324.
[7]
D. Koller, M. Sahami, Toward optimal feature selection, in: Proceedings of the 13th International Conference on Machine Learning, 1996, pp. 284-292.
[8]
I. Guyon, A. Elisseeff, An introduction to variable and feature selection, J. Mach. Learn. Res., 3 (2003) 1157-1182.
[9]
K. Kira, L. Rendell, A practical approach to feature selection, in: Proceedings of the 9th International Conference on Machine Learning, 1992, pp. 249-256.
[10]
J. Weston, S. Mukherjee, O. Chapelle, Feature selection for SVMs, Adv. Neural Inf. Process. Syst., 13 (2000) 668-674.
[11]
O. Chapelle, V. Vapnik, O. Bousquet, S. Mukherjee, Choosing multiple parameters for support vector machines, Mach. Learn., 46 (2002) 131-159.
[12]
Y. Sun, S. Todorovic, S. Goodison, Local learning based feature selection for high dimensional data analysis, IEEE Trans. Pattern Anal. Mach. Intell., 32 (2010) 1-18.
[13]
L. Zhou, L. Wang, C. Chen, Feature selection with redundancy-constrained class separability, IEEE Trans. Neural Netw., 21 (2010) 853-858.
[14]
R. Gilad-Bachrach, A. Navot, N. Tishby, Margin based feature selection theory and algorithms, in: Proceedings of the 21st International Conference on Machine Learning, 2004, pp. 43-50.
[15]
Z. Zhao, H. Liu, Spectral feature selection for supervised and unsupervised learning, in: Proceedings of the 24th International Conference on Machine Learning, 2007, pp. 1151-1157.
[16]
Y. Saeys, I. Inza, P. Larranaga, A review of feature selection techniques in bioinformatics, Bioinformatics, 23 (2007) 2507-2517.
[17]
I. Guyon, S. Gunn, M. Nikravesh, L. Zadeh, Springer, 2006.
[18]
L. Song, A. Smola, A. Gretton, J. Bedo, K. Borgwardt, Supervised feature selection via dependence estimation, in: Proceedings of the 24th International Conference on Machine Learning, 2007, pp. 823-830.
[19]
J. Dy, C. Brodley, Feature Selection for unsupervised learning, J. Mach. Learn. Res., 5 (2004) 845-889.
[20]
D. Cai, C. Zhang, X. He, Unsupervised feature selection for multi-cluster data, in: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2010, pp. 333-342.
[21]
J. Tang, H. Liu, Unsupervised feature selection for linked social media data, in: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2012, pp. 904-912.
[22]
Z. Zhao, H. Liu, Semi-supervised feature selection via spectral analysis, in: Proceedings of the SIAM International Conference on Data Mining, 2007, pp. 641-646.
[23]
Z. Xu, R. Jin, M. Lyu, I. King, Discriminative semi-supervised feature selection via manifold regularization, IEEE Trans. Neural Netw., 21 (2010) 1033-1047.
[24]
G. John, R. Kohavi, K. Pfleger, Irrelevant features and the subset selection problem, in: Proceedings of the 11th International Conference on Machine Learning, 1994, pp. 121-129.
[25]
A. Freitas, Understanding the crucial role of attribute interaction in data mining, Artif. Intell. Rev., 16 (2001) 77-199.
[26]
A. El Akadi, A. Amine, A. El Ouardighi, D. Aboutajdine, A two-stage gene selection scheme utilizing MRMR filter and GA wrapper, Knowl. Inf. Syst., 26 (2011) 487-500.
[27]
A.J. Talln-Ballesteros, C. Hervs-Martnez, J.C. Riquelme, R. Ruiz, Feature selection to enhance a two-stage evolutionary algorithm in product unit neural networks for complex classification problems, Neurocomputing, 114 (2012) 107-117.
[28]
P. Bermejo, L. de la Ossa, J.A. Gmez, J.M. Puerta, Fast wrapper feature subset selection in high-dimensional datasets by means of filter re-ranking, Knowl. Based Syst., 25 (2012) 35-44.
[29]
J. Bernier, J. Ortega, E. Ros, I. Rojas, A. Prieto, A quantitive study of fault tolerance noise immunity and generalization ability of MLPs, Neural Comput., 12 (2000) 2941-2964.
[30]
J. Bernier, J. Ortega, I. Rojas, A. Prieto, Improving the tolerance of multilayer perceptrons by minimizing the statistical sensitivity to weight deviations, Neurocomputing, 31 (2000) 87-103.
[31]
X. Zeng, D. Yeung, Hidden neuron pruning of multilayer perceptrons using a quantified sensitivity measure, Neurocomputing, 69 (2006) 825-837.
[32]
X. Zeng, J. Shao, Y. Wang, S. Zhong, A sensitivity-based approach for pruning architecture of Madalines, Neural Comput. Appl., 18 (2009) 957-965.
[33]
D. Yeung, W. Ng, D. Wang, Localized generalization error model and its application to architecture selection for radial basis function neural network, Neural Netw., 18 (2007) 1294-1305.
[34]
D. Yeung, P. Chan, W. Ng, Radial basis function network learning using localized generalization error model, Inf. Sci., 179 (2009) 3199-3217.
[35]
J. Yang, X. Zeng, S. Zhong, S. Wu, Effective neural network ensemble approach for improving generalization performance, IEEE Trans. Neural Netw. Learn. Syst., 24 (2013) 878-887.
[36]
I. Yeh, W. Cheng, First and second order sensitivity analysis of MLP, Neurocomputing, 73 (2010) 2225-2233.
[37]
K. Shen, C. Ong, X. Li, E. Wilder-Smith, Feature selection via sensitivity analysis of SVM probabilistic outputs, Mach. Learn., 70 (2008) 1-20.
[38]
J. Yang, C. Ong, Feature selection for support vector regression using probabilistic prediction, in: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2010, pp. 343-352.
[39]
J. Moody, C. Darken, Fast learning in networks of locally tuned processing units, Neural Comput., 1 (1989) 281-294.
[40]
J. Park, I. Sandberg, Approximation and radial basis function networks, Neural Comput., 5 (1993) 305-316.
[41]
C. Bishop, Improving the generalization properties of radial basis function neural networks, Neural Comput., 3 (1991) 579-581.
[42]
A. Asuncion, D. Newman, UCI machine learning repository. http://www.ics.uci.edu/mlearn/MLRepository.html
[43]
C.-C. Chang, C.-J. Lin, LIBSVM: a library for support vector machines. http://www.csie.ntu.edu.tw/cjlin/libsvm
[44]
X. Zhu, X. Li, S. Zhang, C. Ju, X. Wu, Robust joint graph sparse coding for unsupervised spectral feature selection, IEEE Trans. Neural Netw. Learn. Syst., 28 (2017) 1263-1275.
[45]
J. Sharma, A. Mishra, K. Saxena, S. Kumar, A hybrid technique for license plate recognition based on feature selection of wavelet transform and artificial neural network, in: 2014 International Conference on Reliability Optimization and Information Technology (ICROIT), 2014, pp. 347-352.
[46]
S. Zhang, X. Li, M. Zong, X. Zhu, R. Wang, Efficient kNN classification with different numbers of nearest neighbors, IEEE Trans. Neural Netw. Learn. Syst., PP (2017) 1-12.
[47]
J.C. Ang, A. Mirzal, H. Haron, H.N.A. Hamed, Supervised, unsupervised, and semi-supervised feature selection: a review on gene selection, IEEE/ACM Trans. Comput. Biol. Bioinf., 13 (2016) 971-989.
[48]
S. Zhang, C. Lei, Y. Fang, Y. Li, R. Hu, X. Hu, Unsupervised feature selection via local structure learning and self-representation, in: 2017 IEEE International Conference on Big Knowledge (ICBK), 2017, pp. 297-302.
[49]
M. Xu, H. Chen, P.K. Varshney, Dimensionality reduction for registration of high-dimensional data sets, IEEE Trans. Image Process., 22 (2013) 3041-3049.
[50]
Y. Yuan, Q. Sun, Multiset canonical correlations using globality preserving projections with applications to feature extraction and recognition, IEEE Trans. Neural Netw. Learn. Syst., 25 (2014) 1131-1146.
[51]
Y. Lu, L. Zhang, B. Wang, J. Yang, Feature ensemble learning based on sparse autoencoders for image classification, in: International Joint Conference on Neural Networks (IJCNN), 2014, pp. 1739-1745.
- A feature selection approach based on sensitivity of RBFNNs
Recommendations
Designing RBFNNs using prototype selection
MCPR'10: Proceedings of the 2nd Mexican conference on Pattern recognition: Advances in pattern recognitionPerformance and accuracy of a neural network are strongly related to its design. Designing a neural network involves topology (number of neurons, number of layers, number of synapses between layers, etc.), training synapse weights, and parameter ...
Unsupervised feature selection method based on sensitivity and correlation concepts for multiclass problems
Feature selection is the problem of eliminating the features which are irrelevant and/or redundant. It can also be assumed as the problem of selecting a small subset of features which are necessary and sufficient to describe the target concept. In this ...
Feature selection via sensitivity analysis of SVM probabilistic outputs
Feature selection is an important aspect of solving data-mining and machine-learning problems. This paper proposes a feature-selection method for the Support Vector Machine (SVM) learning. Like most feature-selection methods, the proposed method ranks ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © Elsevier B.V.
Publisher
Elsevier Science Publishers B. V.
Netherlands
Publication History
Published: 31 January 2018
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 18 Feb 2025