Abstract
In this paper we propose a simple unsupervised approach to learning higher order features. This model is based on the recent success of lightweight approaches such as SOMNet and PCANet to the challenging task of image classification. Contrary to the more complex deep learning models such as convolutional neural networks (CNNs), these methods use naive algorithms to model the input distribution. Our endeavour focuses on the self-organizing map (SOM) based method and extends it by incorporating a competitive connection layer between filter learning stages. This simple addition encourages the second filter learning stage to learn complex combinations of first layer filters and simultaneously decreases channel depth. This approach to learning complex representations offers a competitive alternative to common deep learning models whilst maintaining an efficient framework. We test our proposed approach on the popular MNIST and challenging CIFAR-10 datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aly, S.: Learning invariant local image descriptor using convolutional mahalanobis self-organising map. Neurocomputing 142, 239–247 (2014)
Bruna, J., Mallat, S.: Invariant scattering convolution networks. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1872–1886 (2013)
Chan, T.H., Jia, K., Gao, S., Lu, J., Zeng, Z., Ma, Y.: PCANet: a simple deep learning baseline for image classification? IEEE Trans. Image Process. 24(12), 5017–5032 (2015)
Cireşan, D., Meier, U., Masci, J., Schmidhuber, J.: A committee of neural networks for traffic sign classification. In: International Joint Conference on Neural Networks (IJCNN), pp. 1918–1921. IEEE (2011)
Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 215–223. JMLR (2011)
Coates, A., Ng, A.Y.: The importance of encoding versus training with sparse coding and vector quantization. In: International Conference on Machine Learning (ICML), pp. 921–928. ACM (2011)
Coates, A., Ng, A.Y.: Selecting receptive fields in deep networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 2528–2536. Curran Associates, Inc. (2011)
Cordel, M.O., Antioquia, A.M.C., Azcarraga, A.P.: Self-organizing maps as feature detectors for supervised neural network pattern recognition. In: Hirose, A., Ozawa, S., Doya, K., Ikeda, K., Lee, M., Liu, D. (eds.) ICONIP 2016. LNCS, vol. 9950, pp. 618–625. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46681-1_73
Csurka, G., Dance, C., Fan, L., Willamowski, J., Bray, C.: Visual categorization with bags of keypoints. In: Workshop on Statistical Learning in Computer Vision, European Conference on Computer Vision (ECCV), pp. 1–22. Springer, Berlin (2004)
Culurciello, E., Jin, J., Dundar, A., Bates, J.: An analysis of the connections between layers of deep neural networks. arXiv preprint arXiv:1306.0152 (2013)
Dong, L., He, L., Kong, G., Zhang, Q., Cao, X., Izquierdo, E.: CUNet: a compact unsupervised network for image classification. arXiv preprint arXiv:1607.01577 (2016)
Dundar, A., Jin, J., Culurciello, E.: Convolutional clustering for unsupervised learning. arXiv preprint arXiv:1511.06241 (2015)
Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: LIBLINEAR: a library for large linear classification. J. Mach. Learn. Res. 9(Aug), 1871–1874 (2008)
Hankins, R., Peng, Y., Yin, H.: SOMNet: unsupervised feature learning networks for image classification. In: International Joint Conference on Neural Networks (IJCNN), pp. 1221–1228. IEEE (2018)
Jarrett, K., Kavukcuoglu, K., LeCun, Y., et al.: What is the best multi-stage architecture for object recognition? In: International Conference on Computer Vision (ICCV), pp. 2146–2153. IEEE (2009)
Kannala, J., Rahtu, E.: BSIF: Binarized statistical image features. In: International Conference on Pattern Recognition (ICPR), pp. 1363–1366. IEEE (2012)
Kohonen, T.: Self-organized formation of topologically correct feature maps. Biol. cybern. 43(1), 59–69 (1982)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, Citeseer (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 1097–1105. Curran Associates, Inc. (2012)
Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2169–2178. IEEE (2006). https://doi.org/10.1109/CVPR.2006.68
LeCun, Y., et al.: Handwritten digit recognition with a back-propagation network. In: Advances in Neural Information Processing Systems (NIPS), pp. 396–404. Morgan-Kaufmann (1990)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Lee, H., Grosse, R., Ranganath, R., Ng, A.Y.: Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In: International Conference on Machine Learning (ICML), pp. 609–616. ACM (2009)
Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)
Lin, T.H., Kung, H.: Stable and efficient representation learning with nonnegativity constraints. In: International Conference on Machine Learning (ICML), pp. 1323–1331. JMLR (2014)
Ng, C.J., Teoh, A.B.J.: DCTNet: a simple learning-free approach for face recognition. In: Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pp. 761–768. IEEE (2015)
Ngiam, J., Chen, Z., Chia, D., Koh, P.W., Le, Q.V., Ng, A.Y.: Tiled convolutional neural networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 1279–1287. Curran Associates, Inc. (2010)
Peng, Y., Yin, H.: Markov random field based convolutional neural networks for image classification. In: Yin, H., et al. (eds.) IDEAL 2017. LNCS, vol. 10585, pp. 387–396. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68935-7_42
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Yang, J., Yu, K., Gong, Y., Huang, T.: Linear spatial pyramid matching using sparse coding for image classification. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1794–1801. IEEE (2009)
Yin, H.: The self-organizing maps: background, theories, extensions and applications. In: Fulcher, J., Jain, L.C. (eds.) Computational Intelligence: A Compendium, vol. 115, pp. 715–762. Springer, Berlin (2008). https://doi.org/10.1007/978-3-540-78293-3_17
Zeiler, M.D., Fergus, R.: Stochastic pooling for regularization of deep convolutional neural networks. arXiv preprint arXiv:1301.3557 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Hankins, R., Peng, Y., Yin, H. (2018). Towards Complex Features: Competitive Receptive Fields in Unsupervised Deep Networks. In: Yin, H., Camacho, D., Novais, P., Tallón-Ballesteros, A. (eds) Intelligent Data Engineering and Automated Learning – IDEAL 2018. IDEAL 2018. Lecture Notes in Computer Science(), vol 11314. Springer, Cham. https://doi.org/10.1007/978-3-030-03493-1_87
Download citation
DOI: https://doi.org/10.1007/978-3-030-03493-1_87
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03492-4
Online ISBN: 978-3-030-03493-1
eBook Packages: Computer ScienceComputer Science (R0)