Abstract
It has been shown that the Deep Belief Network is good at modeling input distribution, and can be trained efficiently by the greedy layer-wise unsupervised learning. Hoglak Lee et al. (2008) introduced a sparse variant of the Deep Belief Network, which applied the Gaussian linear units to model the input data with a sparsity constraint. However, it takes much more weight updates to train the RBM (Restricted Boltzmann Machine) with Gaussian visible units, and the reconstruction error is much larger than training an RBM with binary visible units. Here, we propose another version of Sparse Deep Belief Net which applies the differentiable sparse coding method to train the first level of the deep network, and then train the higher layers with RBM .This hybrid model, combining the advantage of the Deep architecture and the sparse coding model, leads to state-of-the-art performance on the classification of handwritten digits.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Hinton, G.E., Osindero, S., Yee-Whye, T.: A Fast Learning Algorithm for Deep Belief Nets. Neural Computation 18, 1527–1544 (2006)
Olshausen, B.A.: Emergence of Simple-cell Receptive Field Properties by Learning a Sparse Code for Natural images. Nature 381, 607–609 (1996)
Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy Layer-Wise Training of Deep Networks. In: NIPS (2007)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Hinton, G.E.: Training Products of Experts by Minimizing Contrastive Divergence. Neural Computation 14, 1771–1800 (2002)
Lee, H., Ekanadham, C., Ng, A.Y.: Sparse Deep Belief Net Model for Visual Area V2. In: NIPS (2008)
Ranzato, M., Poultney, C., Chopra, S., LeCun, Y.: Efficient Learning of Sparse Representations with an Energy-based Model. In: NIPS (2006)
Ranzato, M., Boureau, Y.-L., LeCun, Y.: Sparse Feature Learning for Deep Belief Networks. In: NIPS (2007)
Welling, M., Rosen Zvi, M., Hinton, G.E.: Exponential Family Harmoniums with an Application to Information Retrieval. In: NIPS (2005)
Teh, Y.W., Welling, M., Osindero, S., Hinton, G.E.: Energy-based Models for Sparse Overcomplete Representations. Journal of Machine Learning Research 4, 1235–1260 (2003)
Lee, H., Battle, A., Raina, R., Ng, A.Y.: Efficient Sparse Coding Algorithm. In: NIPS (2006)
Raina, R., Battle, A., Lee, H., Packer, B., Ng, A.Y.: Self-taught Learning: Transfer Learning from Unlabeled Data. In: ICML (2007)
Bradley, D.M., Bagnell, D.: Differentiable Sparse Coding. In: NIPS (2008)
Larochelle, H., Bengio, Y.: Classification using Discriminative Restricted Boltzmann Machine. In: ICML (2008)
Nair, V., Hinton, G.: Implicit Mixtures of Restricted Boltzmann Machine. In: NIPS (2008)
The MNIST database of handwritten digits, http://yann.lecun.com/exdb/mnist/
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Xie, J., Lu, H., Nan, D., Nengbin, C. (2010). Sparse Deep Belief Net for Handwritten Digits Classification. In: Wang, F.L., Deng, H., Gao, Y., Lei, J. (eds) Artificial Intelligence and Computational Intelligence. AICI 2010. Lecture Notes in Computer Science(), vol 6319. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-16530-6_10
Download citation
DOI: https://doi.org/10.1007/978-3-642-16530-6_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-16529-0
Online ISBN: 978-3-642-16530-6
eBook Packages: Computer ScienceComputer Science (R0)