Abstract
Universum linear discriminant analysis was recently proposed to improve linear discriminant analysis by incorporating Universum information. However, it obtains each of the discriminant directions by just using samples from two classes, while other class samples are considered as Universum. This not only leads to the ignoring of some discriminant information, but also restricts its number of extracted features to at most \(0.5k(k-1)\), where k is the number of classes. To fully explore discriminant information from all classes, this paper studies a novel Universum linear discriminant analysis by considering a unified model that simultaneously uses all classes. Compared to the existing Universum linear discriminant analysis, all Universum information is fully utilized in the proposed model when obtaining each discriminant direction, where the Universum can be self-constructed as well can be given advanced of any types. The constrained “concave-convex" procedure can be used to solve the proposed method, which makes the algorithm convergent to a local minimum. By employing a recursive technique, arbitrary number of discriminant directions can be obtained. Experimental results on real-world benchmark datasets and image datasets illustrate the advantages of the proposed method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability statement
The data that support the findings of this study are openly available in [https://archive.ics.uci.edu/ml/datasets.php], and IMM face and Indian females datasets are available on request from the authors.
References
Hastie, T., Tibshirani, R., Friedman, JH.: et al. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. springer, New York (2009)
Moore, B.C.: Principal component analysis in linear systems: controllability, observability, and model reduction. IEEE Trans. Auto. Control 26(1), 17–32 (1981)
Jolliffe, I.T., Cadima, J.: Principal component analysis: a review and recent developments. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 374(2065), 20150202 (2016)
Fisher, R.A.: The use of multiple measurements in taxonomic problems. Ann. Eugenics 7(2), 179–188 (1936)
Fukunaga, K.: Introduction to Statistical Pattern Recognition, 2nd edn. Academic Press, New York (1991)
Vapnik, V.: Statistical Learning Theory, Wiley-Interscience, (1998)
Vapnik, V.: Estimation of Dependencies Based on Empirical Data (Information Science and Statistics). Springer-Verlag, New York (2006)
Zhang, D., Wang, J., Si, L.: Document clustering with Universum. In: Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, PP. 873–882 (2011)
Shen, C., Wang, P., Shen, F., et al.: \(\cal{U}\)Boost: Boosting with the Universum. IEEE Trans. Pattern Anal. Mach. Intell. 34(4), 825–832 (2011)
Liu, C.L., Hsaio, W.H., Lee, C.H., et al.: Semi-supervised text classification with Universum learning. IEEE Trans. Cyber. 46(2), 462–473 (2015)
Richhariya, B., Tanveer, M., Rashid, A.H., et al.: Diagnosis of Alzheimer’s disease using Universum support vector machine based recursive feature elimination (USVM-RFE). Biomed. Signal Process. Control 59, 101903 (2020)
Weston, J., Collobert, R., Sinz, F.: et al. Inference with the Universum. In: Proceedings of the 23rd International Conference on Machine Learning, PP. 1009–1016 (2006)
Qi, Z., Tian, Y., Shi, Y.: Twin support vector machine with Universum data. Neural Netw. 36, 112–119 (2012)
Dhar, S., Cherkassky, V.: Development and evaluation of cost-sensitive Universum-SVM. IEEE Trans. Cyber. 45(4), 806–818 (2014)
Richhariya, B., Tanveer, M.: A reduced Universum twin support vector machine for class imbalance learning. Pattern Recogn. 102, 107150 (2020)
Moosaei, H., Bazikar, F., Ketabchi, S., et al.: Universum parametric-margin \(\nu \)-support vector machine for classification using the difference of convex functions algorithm. Appl. Intell. 52(3), 2634–2654 (2022)
Dhar, S., Cherkassky, V., Shah, M.: Multiclass learning from contradictions. Adv. Neural Inf. Process. Syst., P. 32 (2019)
Wang, Z., Zhu, Y., Liu, W., et al.: Multi-view learning with Universum. Knowl. Based Syst. 70, 376–391 (2014)
Zhang, D., Wang, J., Wang, F.: et al. Semi-supervised classification with Universum. In: Proceedings of the 2008 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, pp. 323–333 (2008)
Xiao, Y., Wen, J., Liu, B.: A new multi-task learning method with Universum data. Appl. Intell. 51(6), 3421–3434 (2021)
Xiao, Y., Feng, J., Liu, B.: A new transductive learning method with universum data. Appl. Intell. 51(8), 5571–5583 (2021)
Chapelle, O., Agarwal, A., Sinz, F.: et al. An analysis of inference with the Universum. Adv. Neural Inf. Process. Syst., p. 20 (2007)
Chen, S., Zhang, C.: Selecting informative Universum sample for semi-supervised learning. In: Twenty-First International Joint Conference on Artificial Intelligence (2009)
Cherkassky, V., Dhar, S., Dai, W.: Practical conditions for effectiveness of the Universum learning. IEEE Trans. Neural Netw. 22(8), 1241–1255 (2011)
Chen, X., Ma, D.: Universum principal component analysis. n: International Conference on Information Technology and Management Engineering, pp. 236–241 (2014)
Qiu, J., Zhang, Y., Pan, Z.: et al. A novel semi-supervised approach for feature extraction. In: 2016 International Joint Conference on Neural Networks (IJCNN). IEEE , pp. 3765–3770 (2016)
Chen, X., Yin, H., Jiang, F., et al.: Multi-view dimensionality reduction based on Universum learning. Neurocomputing 275, 2279–2286 (2018)
Chen, X.H., Chen, S.C., Xue, H.: Universum linear discriminant analysis. Electronics Lett. 48(22), 1407–1409 (2012)
Yuille, A.L., Rangarajan, A.: The concave-convex procedure. Neural Comput. 15(4), 915–936 (2003)
Heisele, B., Poggio, T., Pontil, M.: Face detecton in still gray images (2000). publications.ai.mit.edu,ai-publications/1500-1999/AIM-1687.ps.Z
Xiang, C., Fan, X.A., Lee, T.H.: Face recognition using recursive Fisher linear discriminant. IEEE Trans. Image Process. 15(8), 2097–2105 (2006)
Stegmann, M.B., Ersboll, B.K., Larsen, R.: FAME-a flexible appearance modeling environment. IEEE Trans. Med. Imag. 22(10), 1319–1331 (2003)
Nordstrøm M.M., Larsen M., Sierakowski J. et al.: The IMM face database-an annotated dataset of 240 face images. 2004, Technical University of Denmark, DTU Informatics, Building 321, Technical Report
Jain, V., Mukherjee, A.: The Indian face database. http://vis-www.cs.umass.edu/ vidit/IndianFaceDatabase/ (2002)
Acknowledgements
This work is supported by the National Natural Science Foundation of China (Nos. 62066012 and 12271131), and the Hainan Provincial Natural Science Foundation of China (Nos. 620QN234 and 120RC449).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in the paper “Recursive Universum linear discriminant analysis”.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Li, CN., Liu, J., Meng, Y. et al. Recursive universum linear discriminant analysis. Optim Lett 18, 1405–1419 (2024). https://doi.org/10.1007/s11590-023-02067-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11590-023-02067-9